Systems and methods for providing context-based data for an industrial automation system

ABSTRACT

A non-transitory computer-readable medium includes instructions that, when executed by processing circuitry, are configured to cause the processing circuitry to receive sensory datasets associated with an industrial automation system from sensors, receive positioning data via an extended reality device associated with a user, determine a first virtual positioning of the user in a virtual coordinate system based on the positioning data, determine a second virtual positioning of an industrial automation system in the virtual coordinate system based on the sensory datasets, determine output representative data to be presented by the extended reality device based on the plurality of sensory datasets and in accordance to the first virtual positioning relative to the second virtual positioning, and instruct the extended reality device to present the output representative data.

BACKGROUND

The disclosure relates generally to providing data to a user performinga task associated with an industrial environment. More specifically,embodiments of the present disclosure are related to systems and methodsfor receiving sensory datasets associated with an industrial automationsystem, determining context information, and presenting outputrepresentative data associated with the sensory datasets in a manner orformat based on the context information.

This section is intended to introduce the reader to various aspects ofart that may be related to various aspects of the present techniques andare described and/or claimed below. This discussion is believed to behelpful in providing the reader with background information tofacilitate a better understanding of the various aspects of the presentdisclosure. Accordingly, it should be noted that these statements are tobe read in this light, and not as admissions of prior art.

Various types of data may be used to assist a user to perform a certaintask. As an example, the user may determine an appropriate action to beperformed based on the data, and/or the data may otherwise guide theuser to perform the task. For instance, the data may provideinstructions to perform an action on a component or device of anindustrial automation system. In some embodiments, the data may bepresented using real-world elements and/or computer-generated or virtualelements, including visual elements, audio elements, haptic elements,and the like. Unfortunately, an amount of data presented to the userand/or the manner in which the data is presented to the user may noteffectively or efficiently assist the user with performing the task.Thus, there is a need to improve how data may be presented to the user.

BRIEF DESCRIPTION

A summary of certain embodiments disclosed herein is set forth below. Itshould be noted that these aspects are presented merely to provide thereader with a brief summary of these certain embodiments and that theseaspects are not intended to limit the scope of this disclosure. Indeed,this disclosure may encompass a variety of aspects that may not be setforth below.

In one embodiment, a non-transitory computer-readable medium includesinstructions that, when executed by processing circuitry, are configuredto cause the processing circuitry to receive sensory datasets associatedwith an industrial automation system from sensors, receive positioningdata via an extended reality device associated with a user, determine afirst virtual positioning of the user in a virtual coordinate systembased on the positioning data, determine a second virtual positioning ofan industrial automation system in the virtual coordinate system basedon the sensory datasets, determine output representative data to bepresented by the extended reality device based on the plurality ofsensory datasets and in accordance to the first virtual positioningrelative to the second virtual positioning, and instruct the extendedreality device to present the output representative data.

In one embodiment, a non-transitory computer-readable medium includesinstructions that, when executed by processing circuitry, are configuredto cause the processing circuitry to receive a plurality of sensorydatasets associated with an industrial automation system from aplurality of sensors, monitor positioning data received via an extendedreality device associated with a user, determine a first virtualpositioning of the user in a virtual coordinate system based on thepositioning data, compare the first virtual positioning to a secondvirtual positioning of an industrial automation system in the virtualcoordinate system, determine a subset of the plurality of sensorydatasets based on the first virtual positioning relative to the secondvirtual positioning, determine output representative data to bepresented by the extended reality device based on the subset of theplurality of sensory datasets, and instruct the extended reality deviceto present the output representative data.

In one embodiment, a method includes receiving, via processingcircuitry, a plurality of sensory datasets associated with an industrialautomation system from a plurality of sensors, receiving, via theprocessing circuitry, a first input indicative of a first virtualpositioning of a user in a virtual coordinate system, determining, viathe processing circuitry, first output representative data to bepresented by an extended reality device based on the plurality ofsensory datasets and in accordance to the first virtual positioning ofthe user relative to a second virtual positioning of an industrialautomation system in the virtual coordinate system, instructing, via theprocessing circuitry, the extended reality device to present the firstoutput representative data, receiving, via the processing circuitry, asecond input indicative of an adjustment of the user from the firstvirtual positioning to a third virtual positioning relative to thesecond virtual positioning of the industrial automation system in thevirtual coordinate system, determining, via the processing circuitry,second output representative data to be presented by the extendedreality device based on the plurality of sensory datasets and inaccordance to the third virtual positioning of the user relative to thesecond virtual positioning of the industrial automation system, andinstructing, via the processing circuitry, the extended reality deviceto present the second output representative data.

DRAWINGS

These and other features, aspects, and advantages of the presentdisclosure will become better understood when the following detaileddescription is read with reference to the accompanying drawings in whichlike characters represent like parts throughout the drawings, wherein:

FIG. 1 is a schematic diagram of a communication network to provide auser with data associated with an industrial automation system, inaccordance with an embodiment of the present disclosure;

FIG. 2 is a schematic diagram of a computing system that may beincorporated in a device or system being used to present data to a user,in accordance with an embodiment of the present disclosure;

FIG. 3 is a flowchart of a method or process for presenting outputrepresentative data based on input sensory datasets and related contextinformation, in accordance with an embodiment of the present disclosure;

FIG. 4 is a flowchart of a method or process for presenting outputrepresentative data based on an adjustment of a priority of inputsensory datasets, in accordance with an embodiment of the presentdisclosure;

FIG. 5 is a flowchart of a method or process for presenting outputrepresentative data based on an intent of a user, in accordance with anembodiment of the present disclosure;

FIG. 6 is a flowchart of a method or process for modifying a defaultsetting in which output representative data is presented, in accordancewith an embodiment of the present disclosure;

FIG. 7 is a flowchart of a method or process for presenting storedinformation associated with previous usage of an extended realitydevice, in accordance with an embodiment of the present disclosure;

FIG. 8 is a schematic diagram of a communicating network to presentoutput representative data based on virtual positionings in a virtualcoordinate system, in accordance with an embodiment of the presentdisclosure;

FIG. 9 is a schematic diagram of a communicating network in whichvarious entities are mapped into a virtual coordinate system, inaccordance with an embodiment of the present disclosure;

FIG. 10 is a schematic diagram of a communicating network in whichvarious entities are mapped into a virtual coordinate system, inaccordance with an embodiment of the present disclosure;

FIG. 11 is a flowchart of a method or process for presenting outputrepresentative data based on virtual positionings in a virtualcoordinate system, in accordance with an embodiment of the presentdisclosure;

FIG. 12 is a schematic diagram of an embodiment of a communicationnetwork in which a user controls a machine in an industrial automationsystem, in accordance with an embodiment of the present disclosure;

FIG. 13 is a flowchart of a method or process for presenting outputrepresentative data based on a positioning of a machine in an industrialautomation system, in accordance with an embodiment of the presentdisclosure;

FIG. 14 is a flowchart of a method or process for presenting feedbackbased on an operation being performed by a machine, in accordance withan embodiment of the present disclosure; and

FIG. 15 is a flowchart of a method or processor for providing atime-based user experience, in accordance with an embodiment of thepresent disclosure.

DETAILED DESCRIPTION

One or more specific embodiments of the present disclosure will bedescribed below. In an effort to provide a concise description of theseembodiments, all features of an actual implementation may not bedescribed in the specification. It should be noted that in thedevelopment of any such actual implementation, as in any engineering ordesign project, numerous implementation-specific decisions must be madeto achieve the developers' specific goals, such as compliance withsystem-related and business-related constraints, which may vary from oneimplementation to another. Moreover, it should be noted that such adevelopment effort might be complex and time consuming, but wouldnevertheless be a routine undertaking of design, fabrication, andmanufacture for those of ordinary skill having the benefit of thisdisclosure.

When introducing elements of various embodiments of the presentdisclosure, the articles “a,” “an,” “the,” and “said” are intended tomean that there are one or more of the elements. The terms “comprising,”“including,” and “having” are intended to be inclusive and mean thatthere may be additional elements other than the listed elements. One ormore specific embodiments of the present embodiments described hereinwill be described below. In an effort to provide a concise descriptionof these embodiments, all features of an actual implementation may notbe described in the specification. It should be noted that in thedevelopment of any such actual implementation, as in any engineering ordesign project, numerous implementation-specific decisions must be madeto achieve the developers' specific goals, such as compliance withsystem-related and business-related constraints, which may vary from oneimplementation to another. Moreover, it should be noted that such adevelopment effort might be complex and time consuming, but wouldnevertheless be a routine undertaking of design, fabrication, andmanufacture for those of ordinary skill having the benefit of thisdisclosure.

The present disclosure is generally directed to a system and method forpresenting data to a user to assist the user to perform a desired task.For example, the user may desire to perform a task in an industrialenvironment (e.g., an industrial automation system), such as to installan industrial component, to perform maintenance on an industrialcomponent, to adjust an operation of an industrial component, to inspectan industrial component, to replace an industrial component, and thelike. The user may utilize a device (e.g., a mobile device, a wearabledevice) to present data that helps with performing the desired task. Insome embodiments, the device may include an extended reality device thatmay present real-world elements and/or virtual elements. For example,the device may use virtual reality, which primarily includes a virtualor computer-generated environment, augmented reality, which may presentvirtual or computer-generated elements to alter a physical world fromperceived senses (e.g., by overlaying virtual elements on top of areal-world surrounding), mixed reality, which combines real-worldsurroundings with virtual elements that may interact with the real-worldsurroundings, and/or other extended reality features. By presentingvirtual elements in addition to real-world elements, the device maybetter guide the user to perform the desired task.

Unfortunately, it may be difficult to present data in a manner thateffectively enables the user to perform the desired task. As an example,there may not be an appropriate amount of data being presented. Forinstance, presenting too much data may overwhelm the user, while notpresenting enough data may hinder the user from completing the desiredtask. Additionally, the manner or format in which the data is presentedmay not effectively assist the user to perform the desired task. Forexample, presenting an excess amount of visual data may obstruct a viewof the user and reduce the user's ability to perform a particular taskfor which increased vision may be important. Further still, thepresented data may not be relevant to the user (e.g., may not be neededto complete the desired task). Therefore, the presented data may not bebeneficial to the user.

Accordingly, it is now recognized that there is a need to improve thepresentation of data to the user to facilitate the user performing thedesired task. Thus, embodiments of the present disclosure are directedto a system and method to present data based on context information,which may include different environmental conditions that may beassociated with the user, the desired task, the device presenting thedata, and so forth. In some embodiments, processing circuitry, such asprocessing circuitry of the device or separate from the device, mayreceive various input data, such as visual data, audio data, hapticdata, gustatory data, and/or olfactory data associated with anenvironment, location data associated with the device, biometric dataassociated with the user, other suitable data, or any combinationthereof. The processing circuitry may also determine relevant contextinformation. Based on the context information, the processing circuitrymay determine a priority and/or an importance of the input data. Theprocessing circuitry may then determine a suitable manner in whichoutput data representative of the input data may be presented based onthe priority. As a result, the processing circuitry may cause the deviceto present the output data to the user in the suitable manner. Indeed,the processing circuitry may receive input data in a first format andpresent output data representative of such input data in a secondformat. As an example, the processing circuitry may cause the device topresent an input haptic data (e.g., a temperature) via output visualdata (e.g., textual indication of the temperature) based on the contextinformation. As another example, the processing circuitry may block thedevice from presenting certain output data, such as output datarepresentative of input data that is determined to be irrelevant to theuser. Thus, the processing circuitry may cause the device to present ordemonstrate the data in a way (e.g., an intuitive manner) that improvesguidance to perform the desired task, thereby increasing a qualityand/or an efficiency of task performance.

Although the present disclosure primarily discusses usage of an extendedreality device to present data to a user, any suitable device may beused to present the data. Further, for purposes of discussion, thepresent disclosure is primarily described with reference to performing atask associated with an industrial environment. However, the techniquesdescribed herein may be implemented in any suitable environment,setting, or application.

With the preceding in mind, FIG. 1 is an embodiment of a communicationnetwork 50 for processing and presenting data to a user. Thecommunication network 50 may include one or more of an extended realitydevice 52, which may present data to the user via real-world elementsand/or virtual elements. For example, the extended reality device 52 mayuse virtual reality, augmented reality, and/or mixed reality techniques.In some embodiments, the extended reality device 52 may include a mobileor wearable device to enable the user to move the extended realitydevice 52 more easily. For instance, the extended reality device 52 mayinclude a headset 54 (e.g., a virtual reality headset, an augmentedreality headset), a wearable device 56 (e.g., a body suit, gloves), atablet 58, a phone 60, another suitable device, or any combinationthereof. In the illustrated embodiment, the communication network 50includes a processing system 62 (e.g., a physical server, a cloudcomputing system) is communicatively coupled to the extended realitydevice 52 via a communication link 64 that may permit data exchangebetween components of the communication network 50. For instance, thecommunication link 64 may include any wired or wireless network that maybe implemented as a local area network (LAN), a wide area network (WAN),and the like. The communication link 64 may receive certain data, suchas sensor data, transmitted by the extended reality device 52, and thecommunication link 64 may transmit such data to the processing system62.

The processing system 62 may then process the data received from theextended reality device 52 via the communication link 64. As an example,the processing system 62 may control various operations of the extendedreality device 52 based on the received data. For instance, as furtherdescribed herein, the processing system 62 may instruct the extendedreality device 52 to present certain data in response. In additionallyembodiments, the processing system 62 may provide data to be presentedby the extended reality device 52. For instance, the processing system62 may provide image data, audio data, haptic feedback, and the like, tothe extended reality device 52, and the extended reality device 52 maycontrol the presentation of such data received from the processingsystem 62 without additional instructions from the processing system 62.Although the processing system 62 is separate from the extended realitydevice 52 in the illustrated communication network 50, the processingsystem 62 may be a part of the extended reality device 52 in otherembodiments of the communication network 50. In this way, the extendedreality device 52 may process received data without transmitting thereceived data to another component (e.g., via the communication link 64)for processing.

Furthermore, the communication network 50 may include a database 68,which may be communicatively coupled to the processing system 62 and/orto the extended reality device 52 via the communication link 64. As anexample, the database 68 may store certain information and/or content,and the extended reality device 52 and/or the processing system 62 mayretrieve such information and/or content from the database 68 (e.g., topresent to the user). As another example, the extended reality device 52and/or the processing system 62 may transmit data to the database 68 forstorage and enable such data to be retrieved at a later time.

In some embodiments, the communication network 50 may assist a localuser 70 who may be physically located in an industrial automation system72. For example, the local user 70 may be an operator and/or atechnician who is interacting with an industrial automation component 74of the industrial automation system 72, such as to install, inspect,monitor, study, repair, perform maintenance on, adjust an operation of,and/or to replace the industrial automation component 74. The industrialautomation component 74 may include a variety of different components,such as a controller, a drive, a motor, a sensor, a conveyor, aninput/output (I/O) module, a motor control center, a human machineinterface (HMI), a user interface, contactors, a starter, a relay, aprotection device, a switchgear, a compressor, a network switch (e.g.,an Ethernet switch), a scanner, a gauge, a valve, a flow meter, and soforth. The local user 70 may utilize the extended reality device 52 toperform an action within the industrial automation system 72. Forinstance, the local user 70 may wear or otherwise possess the extendedreality device 52 to present data that helps the local user 70 performvarious tasks.

By way of example, the extended reality device 52 may receive (e.g.,collect) first input data 76 associated with the industrial automationsystem 72. In certain embodiments, the first input data 76 may includesensory datasets. As described herein, sensory datasets may include anyinformation that may be observed by a person (e.g., the local user 70)and that is converted into a digital format. For instance, the sensorydatasets may include visual data, audio data, haptic data, biometricdata, gustatory data, and/or olfactory data. Visual data may include anappearance, a position, an orientation, and/or a movement of variousentities, such as of the local user 70, the industrial automationcomponent 74, and/or the industrial automation system 72. Such visualdata may indicate an identity of local user 70, a location (e.g.,position, orientation) of the local user 70 (e.g., within the industrialautomation system 72, relative to the industrial automation component74), a layout of the industrial automation system 72, an identity of theindustrial automation component 74, an operation of the industrialautomation component 74, a location (e.g., position, orientation) of theindustrial automation component 74, and the like. Audio data may includesounds emitted from the local user 70, the industrial automation system72, and/or the industrial automation component 74 and may indicate anactivity performed by the local user 70, a location of the local user70, an operating status of the industrial automation system 72 and/orthe industrial automation component 74, and the like. Haptic data mayinclude information that may be perceived by touch, such as atemperature of the industrial automation system 72 and/or of theindustrial automation component 74, a structure (e.g., a rigidity, astrength, a material) of the industrial automation component 74, and thelike. Biometric data may include heartbeat data, temperature data,retina reading, facial data (e.g., facial expression data), fingerprintdata, brainwave data of the local user 70, and the like. Such data mayindicate a status of the local user 70, the industrial automationcomponent 74, and/or the industrial automation system 72. Gustatory datamay include tastes or flavors present in the industrial automationsystem 72, such as a taste experienced by a user of the extended realitydevice 52. Olfactory data may include smells, such as smells in theindustrial automation system 72, and may further indicate a status ofthe local user 70, of the industrial automation system 72, and/or of theindustrial automation component 74.

The extended reality device 52 and/or processing system 62 may includeand/or may be communicatively coupled to one or more sensors 78 (e.g., aphysical sensor, a virtual sensor) that may detect various operatingparameters. The sensor(s) 78 may transmit the first input data 76indicative of the operating parameters to the extended reality device52. As an example, the sensor(s) 78 may include an image sensor (e.g., adigital camera, a thermal camera, an infrared camera, an x-ray sensor, aranging sensor, electromagnetic radiation (or light) sensors such as avisual light sensor, an infrared sensor, an ultraviolet sensor, anultrasound imaging system or sensor, a thermal sensor, a laser sensor, aLight Detection and Ranging (LiDAR) sensor), a microphone, a hapticsensor (e.g., a thermal sensor, a pressure sensor), a movement sensor(e.g., an accelerometer), a biometric sensor (e.g., a brain wave sensor,a heart rate monitor, a respiratory rate sensor, a blood pressuremonitor), a gustatory sensor (e.g., an electronic tongue), a smell orodor sensor (e.g., an electronic nose), another suitable sensor, or anycombination thereof. The sensor(s) 78 may also be included on one ormore devices worn by a user (e.g., the local user 70) that arecommunicatively coupled to the extended reality device 52 and/or theprocessing system 62, and the devices may be different than the extendedreality device 52. In other words, the sensor(s) 78 may be included in adevice that does not provide an extended reality experience to the localuser 70 in itself. For example, the sensor(s) 78 may be included in afitness tracker or a smart watch that provides the first input data 76or a portion of the first input data 76 to the extended reality device52 and/or the processing system 62. Indeed, the sensor(s) 78 may beincluded on devices worn by or used by the local user 70. Furthermore,it should be noted that, in some embodiments, the sensor(s) 78 may becommunicatively coupled to the processing system 62 without beingcommunicatively coupled to the extended reality device 52. According, insuch embodiments, the first input data 76 collected by the sensor(s) 78may be provided directly to the processing system 62.

Additionally, the first input data 76 may include a user input. In anexample, the user input may directly indicate sensory datasets, such asan identifier of the local user 70, a location of the local user 70,information associated with the industrial automation component 74, andthe like. In some embodiments, the user may interact with a feature ofthe extended reality device 52 to transmit the user input. Additionally,the user may interact with a feature of another component (e.g., of theindustrial automation component 74) to cause the user input to betransmitted to the extended reality device 52.

The processing system 62 may receive the first input data 76 transmittedto the extended reality device 52 and process the first input data 76.For instance, the processing system 62 may determine first output data80 to be transmitted or output based on the first input data 76. As anexample, the processing system 62 may determine first output data 80that is relevant or important to enable and/or assist the local user 70to perform a task. To this end, the processing system 62 may determinecontext information associated with the first input data 76 toprioritize the first input data 76 and determine the suitable firstoutput data 80. The context information may include environmentalconditions associated with the local user 70 and/or the industrialautomation system 72. In certain embodiments, the processing system 62may associate received first input data 76 with respective categories,and the processing system 62 may determine a priority of the categoriesbased on the context information. For example, the processing system 62may query or retrieve a list, such as a list created by an end user, amanufacturer, and the like, the list defining different possiblepriorities of the categories corresponding to different contextinformation. The processing system 62 may then determine correspondingfirst output data 80 based on the priority.

The first output data 80 may be presented in a different format than theformat of the corresponding first input data 76. In an example, thefirst input data 76 may include image data of the industrial automationcomponent 74. However, the processing system 62 may determine that firstoutput data 80 corresponding to the image data of the industrialautomation component 74 may include audio data associated withindustrial automation component 74, such as an instruction, anidentifier, a manufacturer, and/or a specification associated with theindustrial automation component 74. Accordingly, in this example, audiodata of the first output data 80 may be generated based on receivedvisual data of the first input data 76. In response to determining thefirst output data 80 to be presented, the processing system 62 may causethe extended reality device 52 to present the first output data 80 tohelp the local user 70 perform the task. In some embodiments, the firstoutput data 80 may include instructions (e.g., textual instructions,audio instructions) that may guide the local user 70, certain sensorydatasets that may not be easily detectable by the local user 70, and/orinformation that may cause the local user 70 to focus on a certainaspect of the industrial automation component 74. Indeed, the processingsystem 62 may present the first output data 80 in an intuitive manner tobetter help the user by selecting the information included in the firstoutput data 80, the format(s) in which the first output data 80 ispresented, and/or the position (e.g., spatial position relative to thelocal user 70) in which the first output data 80 is presented. Forexample, the processing system 62 may guide the user to a targetlocation (e.g., toward the industrial automation component 74, toward analarm or a sound, away from a certain area within the industrialautomation system 72) by determining the target location relative to thecurrent location and/or orientation of the user and presenting audiodata (e.g., spoken instructions, a beacon sound) and/or haptic feedbackin a spatial position based on the target location relative to thecurrent location and/or orientation of the user in order to enable thelocal user 70 to navigate to the target location more easily (e.g., byorienting the local user 70 toward the target location).

In additional embodiments, the communication network 50 may assist aremote user 82 who may be remotely located relative to the industrialautomation system 72. For example, the remote user 82 may utilize theextended reality device 52 to remotely inspect the industrial automationsystem 72, such as to determine a current status of the industrialautomation component 74. Therefore, the remote user 82 may completevarious actions with respect to the industrial automation system 72without having to be physically present within the industrial automationsystem 72, thereby reducing a cost (e.g., transportation costs,equipment costs), a time expenditure (e.g., time of transportation ormovement), and/or a complexity (e.g., coordinating schedules forcomponent operations or system operations) associated with performingthe actions. In such embodiments, the context information may include anenvironmental condition associated with the remote user 82. In furtherembodiments, the remote user 82 may include any suitable user, such asuser who is at a first location in the industrial automation system 72separate from a second location of the local user 70. For instance, thelocal user 70 and the remote user 82 may be adjacent to differentindustrial automation components 74 that are located in different areasof the industrial automation system 72. As an example, the remote user82 may be greater than a threshold distance away from the local user 70,an industrial automation component 74, the industrial automation system72, and so forth. Thus, the remote user 82 may not experience certainaspects experienced by the local user 70 without use of thecommunication network 50.

The processing system 62 may therefore receive the first input data 76,which may include various sensory datasets associated with theindustrial automation system 72, and the processing system 62 may causethe extended reality device 52 to present corresponding second outputdata 84 to the remote user 82 based on the first input data 76. By wayof example, since the remote user 82 may not be able to physically viewthe industrial automation system 72, the processing system 62 mayprioritize presenting image data 86 (e.g., a virtual imagerepresentative of the industrial automation system 72) to the remoteuser 82. Indeed, the processing system 62 may cause the extended realitydevice 52 to present second output data 84 that may simulate theenvironment of the industrial automation system 72, such as appearances,machinery sounds, and/or a temperature associated with the industrialautomation system 72, to immerse the remote user 82 in the industrialautomation system 72. Such immersion may better enable the remote user82 to perform a task associated with the industrial automation system72. As such, the second output data 84 presented to the remote user 82may be different from the first output data 80 presented to the localuser 70, even though the same first input data 76 may have beeninitially received. Additionally, the second output data 84 may includesimilar data as the first output data 80 and therefore may providesimilar information to the remote user 82 as that provided to the localuser 70. Thus, the remote user 82 may have access to similar informationto which the local user 70 may have access and better immerse the remoteuser 82 in the industrial automation system 72.

The processing system 62 may also receive second input data 88, whichmay be separate from the first input data 76 associated with theindustrial automation system 72. The processing system 62 may cause theextended reality device 52 to present the second output data 84 based onthe second input data 88. As an example, the second input data 88 mayinclude sensory datasets associated with a remote environment in whichthe remote user 82 is located. For instance, the second input data 88may include audio data indicative that a certain area of the remoteenvironment (e.g., relative to the remote user 82) is loud. As a result,the processing system 62 may cause the extended reality device 52 toavoid presenting the second output data 84 in the form of audio data.Additionally, the second input data 88 may include a user input, such asfrom the remote user 82. The user input may indicate the contextinformation that the processing system 62 may use to determine thecorresponding second output data 84 to be presented. Therefore, theprocessing system 62 may use both the first input data 76 and the secondinput data 88 to determine suitable second output data 84 to bepresented.

Moreover, the remote user 82 may utilize the extended reality device 52to provide guidance to the local user 70 performing a task associatedwith the industrial automation system 72. For this reason, the firstinput data 76 may include information associated with the local user 70,and the second output data 84 presented to the remote user 82 may bebased on such information. As an example, the remote user 82 may beplaced in communication with the local user 70 (e.g., based on anindication provided by the local user 70 and indicative of the localuser 70 needing assistance) to assist the local user 70 with performingthe task, and the processing system 62 may cause the extended realitydevice 52 of the remote user 82 to present the second output data 84 tobetter enable the remote user 82 to assist the local user 70. Indeed,the second output data 84 may include information regarding the localuser 70 (e.g., visual data of the local user 70 and/or of the viewingperspective of the local user 70, audio output from the local user 70,biometric data of the local user 70) to enable the remote user 82 toprovide assistance (e.g., instructions) that may be more personalized ortailored to the local user 70. Furthermore, by providing the remote user82 with the second output data 84 that immerses the remote user 82, theprocessing system 62 may enable the local user 70 to provide assistancethat is more relevant to the situation of the local user 70. The remoteuser 82 may therefore provide better guidance without having to bephysically located at the industrial automation system 72, therebyreducing a cost and/or a time associated with assisting the local user70.

In such embodiments, both the local user 70 and the remote user 82 mayutilize respective extended reality devices 52 to perform a task. Thus,the processing system 62 may cause the extended reality device 52 of thelocal user 70 to present the first output data 80 to the local user 70and cause the extended reality device 52 of the remote user 82 topresent the second output data 84 to the remote user 82. The local user70 and the remote user 82 may therefore utilize the respective outputdata 80, 84 to interact with one another and better perform the task.

Further, in certain embodiments, the processing system 62 may cause theextended reality device 52 used by the remote user 82 to present secondoutput data 84 that is more closely associated with the local user 70.For example, a location of the local user 70 (e.g., within theindustrial automation system 72) may be determined, and the processingsystem 62 may receive first input data 76 that may specifically bemonitored within a threshold distance of the location. As the local user70 moves and changes location, the processing system 62 may receiveupdated first input data 76 associated with a new location of the localuser 70. In this way, the second output data 84 being presented to theremote user 82 may more closely align with the situation of the localuser 70. As an example, such second output data 84 may enable the remoteuser 82 to provide certain instructions, such as guidance of the localuser 70 through the industrial automation system 72. In additionalembodiments, the remote user 82 may manually select a location of whichthe processing system 62 may receive the first input data 76. That is,instead of receiving first input data 76 that is associated with thelocation of the local user 70, the processing system 62 may receivefirst input data 76 that is associated with a location indicated by theremote user 82. In further embodiments, the remote user 82 may be ableto select different local users 70, and the processing system 62 maytrack a particular location of a selected one of the local users 70,receive first input data 76 associated with the particular locationand/or the local user 70, and cause the extended reality device 52 topresent the second output data 84 based on such first input data 76.Therefore, when the local user 70 moves and/or when the remote user 82selects a different local user 70 (e.g., thereby indicating first inputdata 76 of a different location is to be received), the processingsystem 62 may receive updated first input data 76 and cause the extendedreality device 52 to present corresponding updated second output data84.

The processing system 62 may also cause the extended reality device 52to present second output data 84 that is more closely associated with adifferent entity, such as a machine (e.g., a robot) that may bephysically located in the industrial automation system 72 and is beingremotely controlled by the remote user 82 (e.g., to complete a desiredtask). That is, the processing system 62 may determine a location of themachine and may receive first input data 76 that is associated with thelocation. As the location of the machine changes (e.g., based on thecontrol by the remote user 82), the processing system 62 may receiveupdated first input data 76. Additionally, the processing system 62 maycause the extended reality device 52 to present second output data 84specifically associated with the machine. By way of example, theprocessing system 62 may determine a movement, a position, and/or anorientation of the machine, and the processing system 62 may cause theextended reality device 52 to present second output data 84 based on themovement, the position, and/or the orientation. For instance, based on adetermination that a leg of the machine is being moved, the processingsystem 62 may cause the extended reality device 52 to impart hapticfeedback on a corresponding leg of the remote user 82. Further, inresponse to a determination that an arm of the machine is in contactwith an object, the processing system 62 may cause the extended realitydevice 52 to impart haptic feedback on a corresponding arm of the remoteuser 82. The remote user 82 may then use the second output data 84 tomore accurately or desirably control the machine to perform a desiredtask.

In further embodiments, the extended reality device 52 and/or processingsystem 62 may enable interactions between various users, such as betweenmultiple remote users 82 that may each be utilizing a respectiveextended reality device 52. As an example, a respective location, suchas a respective location within a shared virtual coordinate space, ofeach remote user 82 relative to one another may be determined, and arespective projection or visual representation of each remote user 82may be generated via the extended reality devices 52 based on therespective locations. Indeed, the respective projection of each remoteuser 82 may be positioned relative to one another based on therespective location of each remote user 82 within the virtual coordinatespace and a determined movement of each remote user 82. That is, basedon a determination of a remote user 82 moving from a first location to asecond location, the projection corresponding to the remote user 82within the projection may also be moved accordingly. Output data mayalso be presented based on an interaction with the projection. By way ofexample, such interaction may be indicated based on a first location ofa first remote user 82 within the shared virtual coordinate spaceoverlapping with a second location of a second remote user 82 within theshared virtual coordinate space (e.g., the first remote user 82 isshaking hands with the second remote user 82). In response to adetermination of such an interaction, the respective extended realitydevices 52 may present corresponding output data, such as hapticfeedback indicative of the interaction.

In addition to using the input data (e.g., first input data 76, secondinput data 88) for presenting output data (e.g., first output data 80,second output data 84), the processing system 62 may perform otheractions with respect to the input data. In an example, the processingsystem 62 may store the input data (e.g., received input data,categorized input data) and/or corresponding output data in the database68, and the processing system 62 may retrieve the stored data at a latertime, such as to present information associated with the input dataand/or to present the output data at a later time. For instance, theprocessing system 62 may associate the input data and/or the output datawith an identifier, such as an event (e.g., a maintenance event). Basedon a user input indicative of a request to retrieve informationassociated with the identifier, such as to simulate the event, theprocessing system 62 may retrieve the input data and/or the output dataand cause the extended reality device 52 to present the retrieved inputdata and/or output data to a user, thereby simulating a previous statusof the industrial automation system 72. The user may therefore observeor experience a variety of stored information, such as for betterunderstanding of a status history of the industrial automation system72. In additional embodiments, the processing system 62 may processhistorical input data to simulate similar or related situations. Forexample, the processing system 62 may receive input data associated witha first operational status (e.g., during a first operational mode of theindustrial automation component 74) of the industrial automation system72, such as a status that caused a maintenance task or operation to beperformed. In response, the processing system 62 may determine outputdata to be presented based on such input data and associate the outputdata with the first operational status of the industrial automationsystem 72. At a later time, the processing system 62 may simulate thefirst operational status of the industrial automation system 72, such asto train a remote user 82 to perform the maintenance ask, by presentingthe stored output data associated with the first operational status ofthe industrial automation system 72. In this way, the processing system62 may present a more immersive environment to simulate differentsituations associated with the industrial automation system 72 withouthaving to change a real-world status of the industrial automation system72.

FIG. 2 is a schematic diagram of an embodiment of a computing system110, which may be incorporated in the extended reality device 52 and/orthe processing system 62 to perform the techniques described herein. Thecomputing system 110 may include processing circuitry 112, which may beany suitable type of computer processor or microprocessor capable ofexecuting computer-executable code, including but not limited to one ormore central processing units (CPUs), field programmable gate arrays(FPGAs), application-specific integrated circuits (ASICs), programmablelogic devices (PLDs), programmable logic arrays (PLAs), and the like.The processing circuitry 112 may, in some embodiments, include multipleprocessors. The computing system 110 may also include a memory 114 orany suitable articles of manufacture that serve as media to storeprocessor-executable code, data, and the like. The memory 114 may storenon-transitory processor-executable code executed by the processingcircuitry 112 to perform the presently disclosed techniques, such as tocause the output data 80, 84 to be presented.

The computing system 110 may further include a communication component116, which may be a wireless or a wired communication component that mayfacilitate establishing a connection with the communication link 64 toenable communication (e.g., between the extended reality device 52 andthe processing system 62). This wired or wireless communicationcomponent may include any suitable communication protocol includingWi-Fi, mobile telecommunications technology (e.g., 2G, 3G, 4G, 5G, LTE),Bluetooth®, near-field communications technology, and the like. Thecommunication component 112 may include a network interface to enablecommunication via various protocols such as EtherNet/IP®, ControlNet®,DeviceNet®, or any other industrial communication network protocol.

The computing system 110 may additionally include an input/output (I/O)port 118 that may enable the computing system 100 to communicativelycouple to another external device, such as the communication component116 and/or the database 68, via a physical connection (e.g., wiring,cable). Further still, the computing system 110 may include a display120, which may be any suitable image-transmitting component that maydisplay an image. For example, the display 120 may be a display screenthat presents a combination of real-world image data associated with thefield user's physical surroundings with computer-generated image dataassociated with computer-generated elements to supplement the real-worldimage data. In another example, the display 120 may include atransparent display that enables direct viewing of the real-worldsurroundings, and virtual elements may be superimposed onto thetransparent display within the real-world surroundings (e.g., as aheads-up display). In certain embodiments, the computing system 110 mayalso include the sensor(s) 78 that may monitor the first input data 76and/or the second input data 88. For instance, the sensor(s) 78 may becommunicatively coupled to the processing circuitry 112 to enable theprocessing circuitry 112 to perform the techniques described hereinbased on the data received from the sensor(s) 78.

Each of FIGS. 3-7, 11, 13, and 14 discussed below illustrates arespective method or process that may be performed based on the featuresdescribed with respect to FIGS. 1 and 2 . Each method may be performedto assist the local user 70 and/or the remote user 82 to perform a task.In some embodiments, each of the methods may be performed by a singlerespective device or system, such as by the computing system 110 (e.g.,the processing circuitry 112). In the present disclosure, the methodsare described as being performed via the processing system 62. However,in additional embodiments, multiple devices may perform the differentsteps. For example, the processing system 62 may perform the methods incombination with one or more extended reality devices 52. It should alsobe noted that additional steps may be performed with respect to thedescribed methods, and/or certain steps of the depicted methods may beremoved, modified, and/or performed in a different order. Further still,the steps of any of the respective methods may be performed in parallelwith one another, such as at the same time and/or in response to oneanother.

FIG. 3 is a flowchart of an embodiment of a method or process 140 forpresenting output data (e.g., the first output data 80, the secondoutput data 84). At block 142, the processing system 62 may receiveinput sensory datasets (e.g., the first input data 76, the second inputdata 88) associated with the industrial automation system 72 (e.g.,associated with an environment of the industrial automation system 72,of the local user 70, of the remote user 82). In some embodiments, theprocessing system 62 may receive the input sensory datasets from thesensor(s) 78. In additional embodiments, the processing system 62 mayreceive the input sensory datasets via a user input. The input sensorydatasets may include any suitable sensory datasets described herein,such as visual data, audio data, haptic data, biometric data, gustatorydata, and/or olfactory data.

At block 144, the processing system 62 may categorize the input sensorydatasets to form or generate categorized input sensory datasets. Thatis, the processing system 62 may associate the input sensory datasetswith one or more categories (e.g., sensory dataset categories). Forinstance, the processing system 62 may categorize the input sensorydatasets based on the format of the data (e.g., image data, audio data)and/or an object or information pertaining to the data. As an example,image data of the industrial automation component 74 and/or a movementof a user may be associated with a work category; image data associatedwith the view of the user and retina reading of the user may beassociated with a focus category; image data and/or audio dataassociated with the industrial automation system 72 may be associatedwith a system status category; a heart rate, a body temperature, a bloodpressure, and/or a respiratory rate may be associated with a stresscategory; and a temperature of the industrial automation system 72 maybe associated with a virtual category.

At block 146, the processing system 62 may determine context informationassociated with the input sensory datasets. The context information mayinclude an identifier (e.g., a job title or role, a name) of the userassociated with the extended reality device 52, a type of the extendedreality device 52. a location (e.g., of the local user 70, of the remoteuser 82, of the extended reality device 52), a time stamp (e.g., a timeof the day, the week, the month, the year), an activity (e.g., the taskbeing performed with respect to the industrial automation system 72), anissue to be addressed or mitigated (e.g., via performance of theactivity) or any combination thereof.

In some embodiments, the processing system 62 may determine the contextinformation based on received data, such as the received input sensorydatasets (e.g., metadata indicated by the received input sensorydatasets). In an example, the processing system 62 may determine theidentifier of the user based on image data of the user. In anotherexample, the processing system 62 may determine the activity based on animage data of the user and/or the industrial automation system 72 (e.g.,the industrial automation component 74). For instance, the processingsystem 62 may execute instructions to perform image recognition on imagedata, thereby enabling the processing system 62 to determine a taskbeing performed by a user, an item the user is working with (e.g., aparticular tool) or on (e.g., the industrial automation component 74),and/or one or more objects or people included in the image data. In afurther example, the processing system 62 may determine that the user islocated in the industrial automation system 72 (e.g., to determine thatthe user is a local user 70 and not a remote user 82) based on alocation sensor. In yet another example, the processing system 62 mayquery the extended reality device 52 to determine the contextinformation. For instance, the processing system 62 may transmit arequest to the extended reality device 52 to indicate which format ofinformation may be presented by the extended reality device 52. That is,for example, the request may cause the extended reality device 52 toindicate whether the extended reality device 52 may present image data,audio data, haptic feedback, and so forth. Further still, the processingsystem 62 may use speech recognition to determine the contextinformation. By way of example, the processing system may identifyspoken words (e.g., spoken instructions provided by the remote user 82)and determine the context information based on the words.

In additional embodiments, the processing system 62 may determine thecontext information based on a user input. As an example, the user maydirectly indicate that the user is located in the industrial automationsystem 72 and/or a desired task to be completed. In such embodiments,the processing system 62 may also verify the context information, suchas by determining whether the user is qualified to perform the indicateddesired task based on information (e.g., a certification, a level offocus) associated with the user and received via the sensor(s) 78.Indeed, the processing system 62 may perform an action in response tounsuccessful verification of the context information, such as byblocking data from being presented to the user until the unverifiedcontext information has been addressed.

At block 148, the processing system 62 may determine a priority of thecategorized input sensory datasets based on the context information.That is, the processing system 62 may determine an order in which thedifferent categories and therefore the different information associatedwith the categories may be prioritized. In some embodiments, theprocessing system 62 may determine the priority based on informationthat is most relevant, important, and/or beneficial to the userutilizing the extended reality device 52. By way of example, theprocessing system 62 may determine whether the stress category has ahigher or a lower priority than the system status category based on thecontext information. For instance, the local user 70 using the extendedreality device 52 may already know their own stress level and thereforemay not need information related to the stress category, and theprocessing system 62 may therefore prioritize the stress category belowthe system status category. However, it may be beneficial for the remoteuser 82 assisting the local user 70 and using the extended realitydevice 52 to know the stress level of the local user 70 and providebetter instructions accordingly, so the processing system 62 maytherefore prioritize the stress category above the system statuscategory. In additional embodiments, the processing system 62 maydetermine the priority based on a user preference, which may indicatepreferred categories and/or formats of data. Thus, the processing system62 may prioritize the categories based on information associated withthe user, such as an identifier of the user, a user input, and so forth.In certain embodiments, the processing system 62 may use a list (e.g.,stored in the database 68) to set the priority. The list may definerespective priorities of the different categories for different contextinformation. The processing system 62 may therefore compare the contextinformation with the list to select a corresponding respective priorityto associate with each category.

At block 150, the processing system 62 may determine outputrepresentative data to be presented by the extended reality device 52based on the categorized input sensory datasets and the priority (e.g.,to enable the user to intake, observe, or process the information moreeasily and/or effectively). The output representative data maycorrespond to the information included in the input sensory datasets.The processing system 62 may determine a quantity or amount of suchinformation to be presented via the output representative data. That is,the processing system 62 may determine which information (e.g.,information associated with the user, information associated with theindustrial automation system 72) to be presented. The processing system62 may additionally determine a format in which the outputrepresentative data is to be presented. For instance, the processingsystem 62 may determine whether different information or data is to bepresented via image data, text, audio data, haptic feedback, anothersuitable format, or any combination thereof.

By way of example, a greater quantity of information associated withcategorized input sensory datasets of a higher priority may bepresented, and/or such information may be presented at a higherintensity (e.g., larger image data and/or text, louder audio data, morepronounced haptic feedback). However, a lesser quantity of informationassociated with categorized input sensory datasets of a lower prioritymay be presented, and/or such information may be presented via smallertext and/or haptic feedback. In another example, the context informationmay indicate that the extended reality device 52 is unable to presentoutput representative data in a certain format (e.g., via hapticfeedback). Thus, the processing system 62 may determine that informationis to be presented in a different format (e.g., via image data and/oraudio data instead of via haptic feedback) regardless of the priority.By way of example, the processing system 62 may determine components(e.g., the display 120, a speaker, an eccentric rotating mass vibrationmotor) incorporated in the extended reality device 52. At block 152, inresponse to determining the output representative data to be presented,the processing system 62 may cause the extended reality device 52 topresent such output representative data. In other embodiments, theprocessing system 62 may provide the output representative data to theextended reality device 52, and the extended reality device 52 may causethe output representative data to be presented (e.g., via the display120).

To help provide more context for operations included in the process 140,in one example, the context information may indicate that the user is alocal user 70 performing an action with respect to the industrialautomation system 72 (e.g., based on a location of the local user 70, anidentifier of the local user 70, an assigned task, informationassociated with the industrial automation system 72). In response, theprocessing system 62 may prioritize presenting output representativedata corresponding to the system status category over outputrepresentative data corresponding to a stress category associated withthe user. Furthermore, the processing system 62 may determine that theoutput representative data may be presented in a manner that does notobstruct the view of the user to enable the user to view and interactwith the industrial automation system 72. As a result, the processingsystem 62 may cause the extended reality device 52 to present the outputrepresentative data associated with the system status category, such asan overview performance of the industrial automation system 72, via anaudio output, and the processing system 62 may not present outputrepresentative data associated with the stress category, such as a heartrate of the user. In other words, the processing system 62 may providedata determined to be most pertinent to the user in a manner will notobstruct the user's view while also not providing data that is of alower priority (e.g., below a threshold priority level), that wouldobstruct the user's view, or both. However, in response to adetermination (e.g., based on audio data of the input sensory datasets)that the noise level in the industrial automation system 72 is greaterthan a threshold noise level, the processing system 62 may cause theextended reality device 52 to present the output representative dataassociated with the system status category using smaller visual elements(e.g., text) that may be more effectively presented to the user ascompared to using audio data and without obstructing the view of theuser.

In another example, the context information may indicate that the useris a remote user 82 performing an action with respect to the industrialautomation system 72. For this reason, the processing system 62 mayprioritize presenting output representative data associated with thework category over output representative data associated with the focuscategory. Additionally, the processing system 62 may determine that theoutput representative data may be presented via image data withouthindering the remote user's ability to perform a task based on suchcontext information. Thus, the processing system 62 may cause theextended reality device 52 to present the output representative dataassociated with the work category, such as an operation of theindustrial automation component 74, via a visual output, and theprocessing system 62 may present the output representative dataassociated with the focus category, such as whether the remote user 82is viewing the correct section of an image data representing theindustrial automation component 74, via a haptic output (e.g., avibration).

In a further example, the context information may indicate that the useris a remote user 82 assisting a local user 70 with performing an actionassociated with the industrial automation system 72. Thus, theprocessing system 62 may prioritize presenting to the remote user 82output representative data associated with the stress category overoutput representative data associated with the work category, and theprocessing system 62 may determine that the output representative datamay be presented via image data to enable the remote user 82 to viewsuch data more clearly. In response, the processing system 62 may causethe extended reality device 52 to present the output representativedata, such as a heart rate, associated with the stress category via avisual output and to present the output representative data, such as acomponent specification, associated with the work category via an audiooutput. The processing system 62 may also present other information,such as via visual data or indicators, associated with the local user 70and/or the industrial automation system 72 to the remote user 82. Inthis manner, the remote user 82 may modify the presented workinstructions based on the stress level and the visual data and betterguide the local user 70 to perform the action.

In such embodiments, the local user 70 may also be utilizing a separateextended reality device 52 to facilitate performance of the task. Theprocessing system 62 may therefore cause the extended reality device 52of the local user 70 to prioritize and present output representativedata based on the context information indicative that the local user 70is being assisted by the remote user 82. For example, the processingsystem 62 may cause the extended reality device 52 of the local user 70to increase the priority of the work category and may therefore causethe extended reality device 52 of the local user 70 to present workinstructions modified and provided by the remote user 82 via visualdata. The processing system 62 may present the visual data (e.g., text)in a manner that helps the local user 70 perform a task while alsoavoiding obstruction of the view of the local user 70. Indeed, theprocessing system 62 may present different data and/or present data indifferent formats to the local user 70 and the remote user 82 via therespective extended reality devices 52. That is, the first output data80 presented to the local user 70 via the extended reality device 52 ofthe local user 70 may be different from the second output data 84presented to the remote user 82 via the extended reality device 52 ofthe remote user 82. For example, the processing system 62 may cause theextended reality device 52 of the remote user 82 to present the secondoutput data 84 in a first format that enables the remote user 82 toprovide instructions and/or guidance to the local user 70, and theprocessing system 62 may cause the extended reality device 52 of thelocal user 70 to present the first output data 80 in a second formatthat enables the local user 70 to perform actions to complete the task,such as based on the instructions and/or guidance provided by the remoteuser 82.

Further still, the context information may indicate a different statusof the local user 70 and/or the remote user 82. For example, asdescribed above, the processing system 62 may determine a location ofthe local user 70 relative to a target location. The processing system62 may determine output representative data to be presented according tosuch relative location, such as by presenting output representative datain a spatial position that helps guide the local user 70 toward thetarget location. In addition, the processing system 62 may determine astress level of the local user 70 and/or of the remote user 82 and maycause the extended reality device 52 to present output representativedata based on the stress level. By way of example, in response todetermining that the local user 70 and/or the remote user 82 has anincreased stress level (e.g., based on a biometric data value exceedinga threshold value), the processing system 62 may cause the extendedreality device 52 to reduce the intensity (smaller image data and/ortext, quieter audio data, less pronounced haptic feedback, slower speedof a sequence) in which the output representative data is presented totry to reduce the stress level of the local user 70 and/or the remoteuser 82. Moreover, the processing system 62 may determine a target focus(e.g., a target location, target information) of the local user 70and/or the remote user 82. The processing system 62 may determine themanner in which the output representative data is to be presented basedon the target focus. For instance, the processing system may increasethe intensity of certain output representative data (e.g., audio data)to guide the attention of the local user 70 and/or of the remote user 82toward the target focus or otherwise capture the attention of the localuser 70 and/or of the remote user 82. In this way, the processing system62 may cause the output representative data to be presented based on avariety of statuses of the local user 70 and/or the remote user 82 toenable completion of a task.

The processing system 62 may also adjust the priority of the outputrepresentative data based on a desired task to be performed with respectto the industrial automation system 72. As an example, the processingsystem 62 may determine that the remote user 82 is assisting the localuser 70 with performing an inspection of the industrial automationsystem 72, such as based on a current time stamp compared to amaintenance schedule of the industrial automation system 72. Inresponse, the processing system 62 may prioritize the focus category toenable the remote user 82 to determine whether the local user 70 isvisually inspecting the industrial automation system 72 accurately.Thus, the processing system 62 may cause the extended reality device 52of the remote user 82 to present output representative data associatedwith the focus of the local user 70, such as a field of view orperspective of the local user 70, via image data (e.g., video data), inaddition to presenting output representative data associated with thestress and/or visual indicators as text and/or image data.

As another example, the processing system 62 may determine that theremote user 82 is assisting the local user 70 with replacing orrepairing a part of the industrial automation component 74. In response,the processing system 62 may prioritize the system status category overthe stress category. For instance, the processing system 62 may presentthe output representative data associated with the system status of theindustrial automation system 72, such as a fault of a component of theindustrial automation system 72, via image data, in addition topresenting output representative data associated with the workinstructions and the visual indicators as text and/or image data.However, the processing system 62 may present output representative dataassociated with the stress (e.g., a respiratory rate) of the local user70 via haptic feedback, such as tactile pressure points. In certainembodiments, the processing system 62 may cause similar data to bepresented to the local user 70 as well to enable both the remote user 82and the local user 70 to intake the same information and better enablecorrespondence and interaction between the local user 70 and the remoteuser 82. In such cases, while the information provided the remote user82 and the local user 70 may be provided with the same information, theformat(s) of the information provided to the remote user 82 may differfrom the format(s) of the information provided to the local user 70.

In the examples described above, the processing system 62 mayautomatically determine the manner in which the output representativedata may be presented to the user by prioritizing received input sensorydatasets. Indeed, such prioritization of the input sensory datasets mayenable the user to quickly and accurately process relevant informationto determine how to proceed with performing a task. However, in someembodiments, content and/or a format of the output representative datamay be adjustable from a default or initial setting (e.g., a defaultprioritization may be changed to a requested prioritization). Forexample, the user may request to manually set the priority of the inputsensory datasets. Such settings may be stored (e.g., for a profileassociated with the user) and may be retrieved later to present theoutput representative data in a similar manner, such as for the user toperform a similar task.

With this in mind, FIG. 4 is a flowchart of a method or process 170 foradjusting the output representative data that may be presented by theextended reality device 52. For instance, the process 170 may beperformed after certain steps, such as the steps associated with blocks142-148 of the process 140, have been performed to determine a priorityof categorized sensory datasets. At block 172, the processing system 62may receive an indication to adjust a priority of categorized inputsensory datasets. For example, in some embodiments, the processingsystem 62 may receive a user input via the extended reality device 52indicating that the priority of the categorized input sensory datasetsis to be adjusted. In additional embodiments, the processing system 62may receive the indication automatically (e.g., without a user input).

At block 174, in response to receiving the indication to adjust thepriority of the categorized input sensory datasets, the processingsystem 62 may determine updated output representative data to bepresented by the extended reality device 52 based on the categorizedinput sensory datasets and the adjusted priority. That is, theprocessing system 62 may determine updated output representative databased on the categorized input sensory datasets and the receivedindication to adjust the priority of the categorized input sensordatasets. For example, based on the adjustment in priority, certaininput sensory datasets may no longer be presented, certain input sensorydatasets may be presented in a different format (e.g., from beingpresented via visual data to being presented via audio data), and/oradditional input sensory datasets may be presented. At block 176, inresponse to determining the updated output representative data to bepresented, the processing system 62 may cause the extended realitydevice 52 to present the updated output representative data.

In certain embodiments, the processing system 62 may perform the process170 based on the received input sensory datasets indicating that thepriority of categorized input sensory datasets is to be adjusted. By wayof example, the received input sensory datasets may indicate that theuser has moved from a first location within the industrial automationsystem 72 to a second location outside of the industrial automationsystem 72 (e.g., indicative that the user transitioned from being alocal user 70 to being a remote user 82). In response, the processingsystem 62 may update the manner in which the output representative datamay be presented to the user based on the change in location, such as bypresenting more visual data based on a determination that the user isnow a remote user 82. Conversely, if a determination is made that aremote user 82 has become a local user 70, the format of the outputrepresentative data provided by the extended reality device 52 may alsomodified. For instance, relatively less visual data and more audio dataand/or haptic feedback may be provided to the local user 70. In anotherexample, the input sensory data may indicate that a stress level of theuser is increasing (e.g., biometric data, such as the heart rate,associated with the user may exceed a threshold value, audio feedback,such as a tone of voice or key words, provided by the local user 70matching with corresponding audio feedback indicative of increasedstress, certain data, such as image data, indicative that the local user70 is not completing the task), and the processing system 62 may causethe extended reality device 52 to present a lesser quantity or amount ofinformation (e.g., information associated with categorized input sensorydatasets of a lower priority) in response. In this way, the processingsystem 62 may better enable the user to perform the task by accountingfor updates or changes associated with the situation of the user.

In additional embodiments, the processing system 62 may perform theprocess 170 based on a user input, such as a user input transmitted inresponse to an interaction with the extended reality device 52. As anexample, the user input may include information (e.g., a directindication of the location of the user) that the processing system 62may use to determine whether the priority of the categorized inputsensory datasets is to be adjusted. As another example, the user inputmay directly indicate a request to adjust the priority of thecategorized input sensory datasets. That is, for instance, the userinput may directly indicate or define a particular priority of certaincategories and/or that a priority of a particular one of the categoriesis to be increased or decreased. Thus, the processing system 62 mayadjust the priority based on or tailored to the user input regardless ofthe input sensory datasets being received.

Indeed, the user input may indicate a preference of how the outputrepresentative data is to be presented. For example, the preference mayindicate that output representative data associated with the systemstatus category is to be presented via haptic feedback (e.g., ratherthan image data). In certain embodiments, the preference(s) indicated bya user may be associated with a profile of the user. The processingsystem 62 may then store the profile and the associated preference(s)(e.g., in the database 68) and retrieve the profile at a later time topresent the output representative data based on the preference(s). Forexample, the processing system 62 may retrieve the profile based on areceived user input and/or based on other information (e.g., image data,user credentials, identifier) associated with the user. The processingsystem 62 may then determine the preference(s) associated with theretrieved profile and present the output representative data based onthe preference(s). Indeed, the processing system 62 may store multipleprofiles that may each include a different set of preferences, and theprocessing system 62 may retrieve the respective profiles to present theoutput representative data in different manners in accordance with theprofiles. Thus, the processing system 62 may accommodate thepreference(s) of different users without each user having to manuallyre-implement the preference(s) during usage. In additional embodiments,each profile may be associated with other information, such as the taskto be performed, the location of the extended reality device 52, theindustrial automation component 74, and so forth, and the processingsystem 62 may retrieve the profile based on a detection of the otherinformation (e.g., indicated via the context information) and presentthe output representative data based on the retrieved profile.

In further embodiments, the processing system 62 may receive anindication to adjust other manners in which the output representativedata is to be presented by the extended reality device 52. For example,the processing system 62 may receive an indication of a request topresent a certain category of data and/or a specific type of data in aparticular format, such as to present output representative dataassociated with stress (e.g., heart rate data) via text. The processingsystem 62 may also receive an indication of a request to remove certaindata that is being presented or to block a particular format in whichdata is being presented. In other words, the indication may request tofilter out a certain category or type of data (e.g., to block outputrepresentative data associated with stress from being presented) and/orto block data from being presented in a particular manner (e.g., toblock output representative data from being presented via audio data).Such indications may be received automatically (e.g., based on the inputsensory datasets being received) and/or based on a user input, and theprocessing system 62 may update the output representative data to bepresented by the extended reality device 52 based on the indications. Inthis way, the manner in which the processing system 62 may cause theextended reality device 52 to present the output representative data maybe adjustable to accommodate various factors and better present datathat may help the user perform a task.

In addition to causing the extended reality device 52 to present outputrepresentative data based on context information and/or input sensorydata, the processing system 62 may also predict an intent of the userand cause the extended reality device 52 to present outputrepresentative data based on the intent. As described herein, the intentof the user includes an action that the user is to perform to complete adesired task. For example, the intent may include one or more steps forcompleting the desired task. The intent may be associated with a targetoutput or outcome. In other words, successful completion of the intentby the user may produce a desirable result of the industrial automationsystem 72 and/or of the industrial automation component 74. Thus, theprocessing system 62 may present the output representative data tofacilitate achieving the target output.

FIG. 5 is a flowchart of an embodiment of a method or process 200 forpresenting output representative data based on the intent of the user.At block 202, the processing system 62 may receive input sensory dataand/or context information. By way of example, the processing system 62may receive the input sensory data by performing the step described withrespect to block 142 of the method 140, and the processing system 62 mayreceive the context information by performing the step described withrespect to block 146 of the method 140. Such input sensory data mayinclude audio data, vibration data, operating speed data (e.g.,rotational speed of a motor), temperature data, other suitable data, orany combination thereof.

At block 204, the processing system 62 may predict the intent of theuser based on the input sensory data and/or the context information. Insome embodiments, the database 68 may store a list of intents anddifferent input sensory data (e.g., operating parameter values) and/orcontext information associated with each intent of the list of intents.The processing system 62 may select an intent from the list of intentsbased on a match between the received input sensory data and/or contextinformation with corresponding input sensory data and/or contextinformation associated with the intent. In additional embodiments, theprocessing system 62 may determine the desirable task of the user (e.g.,based on the input sensory data and/or the context information) and maydetermine the intent based on the task. For instance, the database 68may store a list of tasks and one or more intents associated with eachtask. Upon selection of the task, the processing system 62 may thenidentify the intent(s) specifically associated with the task and selecta corresponding intent.

As an example, based on a determination that there is an alarm (e.g.,based on received audio data) and that an identity of the user isassociated with a technician (e.g., based on received user input), theprocessing system 62 may determine that the user is to perform amaintenance task or operation. As another example, the processing system62 may compare the input sensory data with respective ranges of valuescorresponding to expected or desirable operations. For instance, theprocessing system may determine that a maintenance task is to beperformed based on one or more of the input sensory data being outsideof respective ranges of values. In certain embodiments, differentmaintenance tasks may be associated with different combinations of inputsensory data being outside of a corresponding range of values (e.g., asdefined via a database table stored in the database 68), and theprocessing system 62 may select a maintenance task based on thecomparison between the input sensory data with the corresponding rangeof values. By way of example, the processing system 62 may determinethat a first maintenance task (e.g., replacement of a motor) is to beperformed based on the operating speed data being below a first range ofvalues and the audio data being below a second range of values (e.g., toindicate the motor is faulty). The processing system 62 may alsodetermine that a second maintenance task (e.g., reducing operation ofthe industrial automation component 74) is to be performed based on theoperating speed being above the first range of values and thetemperature data being above a third range of values (e.g., to indicateoverheating and/or overuse of a motor of the industrial automationcomponent 74). The processing system 62 may then select an intentassociated with the specific maintenance task.

At block 206, the processing system 62 may determine outputrepresentative data to be presented by the extended reality device 52based on the predicted intent to present at least a portion of the inputsensory data. That is, the processing system 62 may determine outputrepresentative data to be presented by the extended reality device 52,such as an amount of information to be presented and/or a format inwhich the output representative data is to be presented. Such outputrepresentative data may facilitate the user with completing an actionassociated with the intent, such as by processing information moreeasily and/or effectively. The processing system 62 may also determinethe output representative data based on the context information toenable completion of the action. For instance, the processing system 62may determine a location of the extended reality device 52 (e.g.,indicative of whether the user is a local user 70 or a remote user 82),an identity of a user utilizing the extended reality device (e.g., todetermine a profile associated with the user and indicative of apreferred format to present the output representative data), availableoutput devices of the extended reality device to present the outputrepresentative data in various formats, and so forth. Indeed, theprocessing system 62 may determine output representative data (e.g., aformat of the output representative data) that enables completion of theaction in view of the received context information. At block 208, theprocessing system 62 may then cause the extended reality device 52 topresent such output representative data (e.g., by transmitting aninstruction to the extended reality device 52, by directly providingtransmitting the output representative data to the extended realitydevice 52).

In certain embodiments, the processing system 62 may determine that theintent of the user has changed and may therefore determine updatedoutput representative data to be presented. By way of example, theprocessing system 62 may determine a target output or result associatedwith completion of the action associated with the intent. The targetoutput may include a target visual output (e.g., an appearance of theindustrial automation system 72 and/or of the industrial automationcomponent 74), a target audio output (e.g., a sound generated viaoperation of the industrial automation system 72 and/or of theindustrial automation component 74), an operating status (e.g., anoperating mode), a performance (e.g., a temperature, a vibration, anoperating speed of the industrial automation system 72 and/or of theindustrial automation component 74), another suitable output, or anycombination thereof. The processing system 62 may determine that theaction has been completed based on a determination that the targetoutput was achieved (e.g., based on received audio data, image data,temperature data, vibration data, operating speed data). In additionalembodiments, the processing system 62 may determine that the action hasbeen completed based on a user input. For example, the user may interactwith the extended reality device 52 to indicate or verify that theaction has been successfully completed, such as by selecting a menuvisually presented to the user and/or by providing audio feedback (e.g.,spoken words).

After determining that the action has been completed, the processingsystem 62 may then predict a subsequent intent of the user. Forinstance, the processing system 62 may determine the subsequent intent(e.g., including an additional action to complete the task) based onupdated input sensory data and/or context information. Additionally, theprocessing system 62 may select the subsequent intent from a set ofintents associated with the identified task stored in the database 68.The processing system 62 may therefore repeat performance of any of thesteps described in blocks 202, 204, 206, 208 in response to adetermination that a previous intent is no longer relevant. Indeed, theoutput representative data being presented may dynamically change as theuser completes various actions. Furthermore, the processing system 62may cause the extended reality device 52 to present outputrepresentative data indicative that the action associated with theintent has been successfully completed. Thus, the processing system 62may inform the user when each action has been completed, such as tonotify the user that updated output representative data is to bepresented to facilitate completion of an additional action.

At block 210, the processing system 62 may receive feedback indicativethat the output representative data being presented by the extendedreality device 52 does not facilitate completing the action associatedwith the intent. The processing system 62 may identify receipt offeedback indicative that the output representative data does notfacilitate completing the action associated with the intent based on thefeedback indicating that the target output was not achieved. Forexample, the processing system 62 may receive user feedback (e.g., amenu selection, a gesture, audio feedback, haptic feedback, biometricdata) from the user. The user feedback may indicate increased stressand/or frustration of the user as well as how the output representativedata does not facilitate completing the action. For instance, the userfeedback may include an increased audio level of the user with spokenkeywords associated with the deficiencies of the output representativedata (e.g., that the output representative data facilitates completionof an action associated with a different intent). In additionalembodiments, the processing system 62 may receive a user input (e.g.,via the extended reality device 52) that directly indicates that theoutput representative data does not facilitate completing the action.

In some embodiments, the processing system 62 may determine that theintent was incorrectly identified (e.g., as indicated by the user input)in response to receiving the feedback. As a result, the processingsystem 62 may perform the step described at block 204 to re-predict theintent of the user and/or by prompting the user to provide the intentvia user input. In additional embodiments, the processing system 62 maydetermine that the intent was correctly predicted (e.g., as indicated bythe user input) in response to receiving the feedback, but the manner inwhich the output representative data is presented does not enablecompletion of the action associated with the intent. As an example, theprocessing system 62 may receive a user input indicative that the amountof information and/or the format of the output representative data doesnot enable the action to be completed. In response, the processingsystem 62 may perform the step described at block 206 to re-determineoutput representative data to be presented by the extended realitydevice 52, such as to present different information and/or to presentthe output representative data in a different format.

At block 212, the processing system 62 may then adjust the outputrepresentative data to be presented by the extended reality device 52based on the received feedback. For instance, in response to determiningthat the intent was incorrectly predicted, the processing system 62 maydetermine a new intent and determine corresponding output representativedata associated with the new intent (e.g., based on information storedin the database 68). In response to determining that the intent wascorrectly predicted and that the output representative data does notenable completion of the action associated with the intent, theprocessing system 62 may adjust how the output representative dataassociated with the intent is presented. In some embodiments, theprocessing system 62 may prompt the user to transmit a user inputindicative of specific information or a format associated withpresentation of the output representative data (e.g., a preferred mannerin which the output representative data is to be presented). Theprocessing system 62 may then cause the extended reality device 52 topresent the output representative data based on the user input. As such,the processing system 62 may adjust how the output representative datais presented to the user to better facilitate completion of the action.

In addition to adjusting output representative data for a single user,the processing system 62 may adjust a default setting in which outputrepresentative data may be presented for multiple users. The defaultsetting refers to a manner in which output representative data may beinitially presented without receiving user input indicative of a requestto change how the output representative data is presented (e.g., asdescribed with respect the method 170 of FIG. 4 ). That is, the defaultsetting may define which information (e.g., corresponding to at least aportion of the input sensory datasets) is to be presented and/or a datapresentation format in which the information is presented. However, theprocessing system 62 may change the default setting based on feedbackindicative that the default setting is not preferred (e.g., does notfacilitate completing a task), such as via user input, thereby changinghow the output representative data may be initially presented to a user.For instance, the processing system 62 may adjust the default setting toavoid receiving subsequent requests (e.g., use inputs) to change how theoutput representative data is presented. In this manner, the adjusteddefault setting of the output representative data may better facilitatecompletion of a task.

With the preceding in mind, FIG. 6 is an embodiment of a method orprocess 230 for modifying the default setting in which outputrepresentative data is presented. At block 232, the processing system 62causes the extended reality device 52 to present output representativedata according to a default setting. For example, respective defaultsettings may be defined (e.g., within the database 68) and associatedwith various priorities and/or different context information ofcategorized input sensory data. Thus, the processing system 62 mayselect a corresponding default setting based on the determined priorityof the categorized input sensory data and/or the determined contextinformation. The processing system 62 may then cause the extendedreality device 52 to present the output representative data in themanner defined by the selected default setting.

At block 234, the processing system 62 may receive feedback to changepresentation of the output representative data. That is, the feedbackmay indicate that the output representative data is to be presented in adifferent manner than that defined by the default setting. In someembodiments, the feedback may include a user input received via theextended reality device 52 (e.g., to indicate a desired or preferredmanner in which the output representative data is to be presented). Forexample, the user input may indicate changing the data presentationformat in which certain information is to be presented, presentingadditional information (e.g., corresponding to a portion of inputsensory datasets not previously presented via the initial outputrepresentative data), or removing certain information (e.g.,corresponding to a portion of input sensory datasets previouslypresented via the initial output representative data) from beingpresented. In additional embodiments, the feedback may indicate that atask or action was not successfully completed (e.g., via the stepperformed at block 210 of the method 200). As an example, the feedbackmay include received sensor data indicating that a target output was notachieved and/or a user input directly indicative that the task or actionwas not completed. Thus, the processing system 62 may determine thatparticular information being provided via the output representative datadoes not facilitate completion of the task, and the processing system 62may adjust the format in which the particular information is beingpresented.

At block 236, the processing system 62 may cause the extended realitydevice 52 to present the output representative data according to thefeedback. In embodiments in which the feedback is received via a userinput that defines a manner in which the output representative data isto be presented, the processing system 62 may cause the extended realitydevice 52 based on the manner defined by the user input. In embodimentsin which the feedback is received based on a determination that a taskor action was not successfully completed, the processing system 62 mayperform a further analysis to determine how the output representativedata is to be presented. By way of example, the processing system 62 maydetermine that the same or a similar task or action was previouslycompleted at a different time (e.g., by a different user), determine themanner in which the output representative data was previously presentedat that time, and present the output representative data in the samemanner as the previously presented output representative data.Additionally, one or more alternative settings may be defined (e.g.,within the database 68) and associated with the default setting. Thealternative setting(s) may also define the manner (e.g., an alternativedata presentation format) in which the output representative data is tobe presented, and the processing system 62 may select a correspondingalternative setting in response to receipt of the feedback. Forinstance, each alternative setting may be associated with a rank ororder number, and the processing system 62 may select the alternativesetting having the next highest rank or order number and cause theextended reality device 52 to present the output representative dataaccording to the alternative setting in response to receiving thefeedback.

At block 238, the processing system 62 may determine whether the defaultsetting is to be adjusted based on the feedback and historical dataassociated with presentation of the output representative data. Forexample, the historical data may indicate that the processing system 62has previously received the same or a similar feedback to change thepresentation of the output representative data. As such, the processingsystem 62 may monitor a number of times in which the same or similarfeedback has been received. The processing system 62 may then comparethe number to a threshold number.

In some embodiments, the database 68 may store information associatedwith changes to the output representative data presented by multipledifferent extended reality devices 52. As an example, the database 68may store profiles of different users who may utilize a differentextended reality device 52, and each profile may include respectivepreferred settings (e.g., defining preferred information to be presentedand/or preferred formats to present the output representative data),which may be input, set, created, or adjusted by the corresponding user.Each preferred setting may be adjusted from a corresponding defaultsetting. The processing system 62 may determine a number of preferredsettings that are different from the corresponding default setting(e.g., indicative that a corresponding number of users changingpresentation of the default settings) and determine whether the defaultsetting is to be modified based on the number. For instance, theprocessing system 62 may determine a number of preferred settings thatindicate operating data of an industrial automation component 74 is tobe presented via audio data instead of via visual data as defined by thedefault setting.

In additional embodiments, the processing system 62 may determinepreferred settings of a single user (e.g., of a profile associated withthe user) and modify the default settings associated with the user. Forinstance, the processing system 62 may determine a preference of theuser during applications or situations associated with certain contextinformation. The processing system 62 may then determine whether thepreference is to be applied during other applications associated withdifferent context information. By way of example, the processing system62 may determine that preferred settings for a profile of a local user70 indicates that operating data of an industrial automation component74 is to be presented via visual data for performing a maintenance task,an inspection task, and a replacement task. As a result, the processingsystem 62 may also determine whether the preferred settings for theprofile of the local user 70 is to be modified to present the operatingdata of the industrial automation component 74 via visual data forperforming other tasks as well, such as an installment task or a taskperformed on a different industrial automation component.

In response to a determination that the default setting should not bemodified based on the feedback and the historical data (e.g., the numberof received feedbacks is below the threshold number), the processingsystem 62 may not change the default setting. As such, the processingsystem 62 may select the default setting and initially presentsubsequent output representative data in the same manner as previouslydefined by the default setting. In certain embodiments, the processingsystem 62 may modify the preferred settings of the local user 70 (e.g.,of the profile associated with the local user 70) from which thefeedback was received to change the presentation of the outputrepresentative data, but the processing system 62 may not change thedefault settings or preferred settings of other users in response to thedetermination that the default setting should not be modified.

However, in response to a determination that the default setting shouldbe modified, the processing system 62 may modify the default setting toreflect the received feedback, as described at block 240. For example,the received feedback may directly indicate an adjustment of thepriority of categorized input sensory data, that output representativedata of a certain category is to be presented in a first format ratherthan in a second format, that specific output representative data is notto be presented, and the processing system 62 may apply the adjustmentto the default setting. For instance, the processing system 62 mayidentify profiles having preferred settings that have not already beenmodified according to the received feedback, and the processing system62 may modify the profiles to update the preferred settings based on thereceived feedback. That is, the processing system 62 may modify theprofiles such that each profile reflects the received feedback (e.g., toinclude the modified default setting). In embodiments in which theprocessing system 62 is adjusting the default settings associated with asingle user, the processing system 62 may prompt the user to confirmthat the default setting is to be adjusted in the manner determined bythe processing system 62. As such, the processing system 62 may alsoverify that the default setting is to be adjusted based on a user input.After modifying the default setting, the processing system 62 maypresent subsequent output representative data in the manner defined bythe modified default setting instead of in the manner previously definedby the unmodified default setting.

In some circumstances, it may be desirable to store, retrieve, andpresent interactions between users (e.g., between a local user 70 and aremote user 82) utilizing the techniques described above to facilitateother users performing the same or a similar task. For example, theprocessing system 62 may be utilized to enable the local user 70 and theremote user 82 to interact with one another (e.g., by communicativelycoupling their respective extended reality devices 52) to complete atask. For instance, the processing system 62 may present the firstoutput data 80 (e.g., in a first data presentation format) to the localuser 70 via the extended reality device 52 of the local user 70 andpresent the second output data 84 (e.g., in a second data presentationformat) to the remote user 82 via the extended reality device 52 of theremote user 82. During presentation of the first output data 80 and thesecond output data 84, the processing system 62 may receive the firstinput data 76 from the local user 70 and the second input data 88 fromthe remote user 82, such input data 76, 88 including different feedbackprovided as interactions (e.g., verbal exchanges) between the local user70 and the remote user 82 and/or such input data 76, 88 including data(e.g., image data) that is directly presented. The interactions betweenthe local user 70 and the remote user 82 may help the same or adifferent local user 70 complete another task and/or help the same or adifferent remote user 82 better help the local user 70 complete anothertask. That is, the interactions, which may not be included in the firstoutput data 80 or the second output data 84 to complete an initial, mayprovide supplemental information that may help completion of asubsequent task. As such, various information associated with theinteraction may be stored, retrieved, and presented for trainingpurposes, such as an instructional video, textual work instructions, andthe like, that can be presented (e.g., without additional input afterinitiating presentation). Although the present disclosure primarilydiscusses usage of interactions between the local user 70 and the remoteuser 82, interactions between any suitable users, such as between localusers 70, between remote users 82, and/or between multiple differenttypes of users, may be used as training content.

FIG. 7 is a flowchart of a method or process 260 for storing aninteraction between the local user 70 and the remote user 82 andretrieving information associated with the interaction for presentation.However, in certain embodiments, a method similar to the method 260 maybe performed for utilizing an interaction between the users. At block262, the processing system 62 may determine a task to be performed basedon context information and input sensory data. For example, theprocessing system 62 may determine a specific maintenance task to beperformed using the techniques described above based on a comparisonbetween the input sensory data (e.g., audio data, vibration data,operating speed data, temperature data) and respective threshold ranges.The processing system 62 may also determine the task to be performedbased on a user input (e.g., provided via the extended reality device52) and/or based on an identifier of a user.

At block 264, the processing system 62 may cause the extended realitydevice 52 to present output representative data based on the contextinformation and the input sensory data using the techniques describedabove. During presentation of the output representative data, theprocessing system 62 may receive remote user feedback and/or local userfeedback, as shown at block 266. As an example, the processing system 62may receive the remote user feedback via the extended reality device 52of the remote user 82, and the remote user feedback may include guidanceprovided by the remote user 82 to assist the local user 70, such asaudio feedback (e.g., spoken words) and/or image feedback (e.g.,modification of the video displayed to the local user 70). In response,the processing system 62 may cause the extended reality device 52 of thelocal user 70 to present the remote user feedback. For instance, theprocessing system 62 may cause the extended reality device 52 to presentaudio data and/or image data corresponding to the received audiofeedback and/or image feedback. The processing system 62 may alsoreceive the local user feedback via the extended reality device 52 ofthe local user 70 and/or the sensor(s) 78, and the local user feedbackmay include reactions of the local user 70 made in response to receiptof the remote user feedback, such as audio feedback, image feedback,biometric data, and the like.

At block 268, the processing system 62 may store the input sensory data,the remote user feedback (e.g., the audio data and/or the image datacorresponding to the remote user feedback), the local user feedback,and/or the output representative data as training content (e.g., in thedatabase 68) for performing the task determined with respect to block262. In some embodiments, the processing system 62 may store suchtraining content in response to a determination that there currently isno previously stored training content associated with performing thetask. In an example, the processing system 62 may determine the task(e.g., based on context information and/or input sensory data),determine whether there is associated training content stored in thedatabase 68, and store the input sensory data, the remote user feedback,the local user feedback, and/or the output representative data inresponse to a determination that there is no training content associatedwith the task stored within the database 68. In another example, theprocessing system 62 may store training content regardless of whetherthere is associated training content stored in the database 68. Forinstance, the processing system 62 may store training content each timethe processing system 62 is in operation (e.g., each time the processingsystem 62 communicatively couples extended reality devices 52 of users)to perform a task. In a further example, the processing system 62 maystore training content based on a user input (e.g., received from thelocal user 70, received from the remote user 82), which may indicate arequest to store the training content.

At block 270, the processing system 62 may receive an indication topresent the training content via an additional extended reality device52. In some embodiments, the processing system 62 may receive theindication via a user input (e.g., by a different user utilizing theadditional extended reality device 52) indicative of a request to assistwith performing the task associated with the training content. Inadditional embodiments, the processing system 62 may receive additionalcontext information and/or additional input sensory data, and theprocessing system 62 may automatically determine that the taskassociated with the training content is to be performed (e.g., withoutreceiving a user input) based on the additional context informationand/or the additional input sensory data. That is, the processing system62 may receive the indication based on a determination that the sametask associated with the training content is to be performed (e.g., by adifferent user utilizing the additional extended reality device 52).

At block 272, in response to receiving the indication, the processingsystem 62 may retrieve the training content (e.g., from the database 68)and cause the additional extended reality device 52 to present the inputsensory data, the remote user feedback, the local user feedback, and/orthe output representative data. In some embodiments, the processingsystem 62 may present the training content in the same format than thatinitially received, presented, and stored. For example, the outputrepresentative data may include a video of the view of the local user70, and the processing system 62 may present the video duringpresentation of the training content. In additional embodiments, theprocessing system 62 may present the training content in a differentformat. As an example, the output representative data may includebiometric data (e.g., of the local user 70) initially presented viaaudio data (e.g., to the remote user 82). However, the processing system62 may present the biometric data via image data during presentation ofthe training content.

The input sensory data, the remote user feedback, and/or the local userfeedback may provide additional information to facilitate completion ofthe task. By way of example, the information may include instructionsgiven by a previous remote user 82 (e.g., to a previous local user 70)and not presented via previous output representative data, and theinstructions may better guide a current local user 70 perform adesirable action. The information may additionally include expectedresults or outputs associated with actions performed by a previous localuser 70, and the expected results may help the current local user 70identify which actions are to be performed to achieve a desired result.Further, the information may help a current remote user 82 determineinstructions to provide to the local user 70 (e.g., whether previouslyprovided instructions successfully enabled a previous local user 70 toperform a desirable action, the impact of previously providedinstructions on the biometric data of a previous local user 70) tocomplete the task. In this manner, the processing system 62 may betterenable completion of the task with use of the training content, such asby supplementing the training content with output representative databeing presented.

Multiple User Collaboration

In some circumstances, it may be desirable for users at differentlocations to interact with one another with respect to the industrialautomation system 72. For example, different locations may includedifferent remote environments (e.g., different geographic locations),different locations within the same environment (e.g., within the sameindustrial automation system 72), and other locations in which the usersmay not be able to physically or directly communicate with one another.However, the users may desire to refer to the same equipment (e.g., thesame industrial automation component 74) or other parts of theindustrial automation system 72. The techniques discussed herein mayimprove the collaboration between users located at different locations,such as by simulating real-life interactions between the users throughpresentation of virtual elements. For example, the processing system 62may cause different output representative data associated with theindustrial automation system 72 to be presented to the remote users inorder to realistically simulate a virtual environment representative ofthe industrial automation system 72. Thus, the processing system 62 mayprovide the users with relevant information that may facilitate userinteractions with respect to the industrial automation system 72, suchas to determine an operation to perform on the industrial automationsystem 72. Furthermore, the processing system 62 may cause outputrepresentative data associated with the users to be presented to oneanother, thereby enabling the users to interact more easily with oneanother, such as by simulating real-life communication between users. Assuch, the processing system 62 may facilitate users performinginteractions with one another and/or with the industrial automationsystem 72.

FIG. 8 is a schematic diagram of an embodiment of a communicationnetwork 290 for enabling multiple user collaboration between a firstremote user 82A and a second remote user 82B. The remote users 82 may belocated at different remote environments 292. For example, the firstremote user 82A may be located in a first remote environment 292A, andthe second remote user 82B may be located in a second remote environment292B, such that the first remote user 82A and the second remote user 82Bmay not physically interact with one another (e.g., speak directly toone another without usage of additional devices). For instance, each ofthe remote environments 292 may include different areas of the same room(e.g., different office cubicles), different rooms of the same building,different geographical locations, and the like. In some embodiments, atleast one of the remote environments 292 may be separate from a portionor section of the industrial automation system 72. Thus, neither of theremote users 82 may be able to interact directly with certain equipment(e.g., industrial automation components 74) of the industrial automationsystem 72. Additionally, at least one of the remote environments 292 maybe a part of the industrial automation system 72. That is, at least oneof the remote users 82 may be local to the industrial automation system72, but remote relative to the other of remote user 82. Furthermore, thecommunication network 290 may include more than two remote users 82remotely collaborating with one another.

Each of the remote users 82 may utilize a respective extended realitydevice 52. For instance, the first remote user 82A may utilize a firstextended reality device 52A, and the second remote user 82B may utilizea second extended reality device 52B. The processing system 62 mayoperate the extended reality devices 52 to enable collaboration betweenthe remote users 82 with one another and/or with the industrialautomation system 72 by causing the extended reality devices 52 topresent output representative data based on position or location. Forexample, the processing system 62 may receive a user input from theextended reality devices 52 indicative of a request to initiate multipleuser collaboration. The user input may, for instance, indicate a requestto present a visual image representative of the other user to enablemultiple user collaboration. In response, the processing system 62 mayuse data associated with the industrial automation system 72 and/or theextended reality devices 52 to present the visual image and otheroutputs to facilitate multiple user collaboration.

In some embodiments, the processing system 62 may determine virtualpositionings of the first remote user 82A, the second remote user 82B,and/or the industrial automation system 72 relative to one anotherwithin a virtual coordinate system, and the processing system 62 maycause the extended reality devices 52 to present output representativedata based on the virtual positionings. Each of the virtual positioningsmay include a position, orientation, and/or pose within the same virtualcoordinate system, and the processing system 62 may use the virtualpositionings to present output representative data that simulatesreal-life physical positionings within a physical environment (e.g., thesame industrial automation system 72). As an example, the processingsystem 62 may determine physical movement of one of the remote users 82(e.g., within their remote environment 292), determine correspondingmovement in the virtual coordinate system, and cause the extendedreality device 52 of each of the remote users 82 to present updatedoutput representative data based on the movement in the virtualcoordinate system.

For instance, the processing system 62 may cause the first extendedreality device 52A to present image data that includes the second remoteuser 82B located (e.g., positioned, oriented) within the industrialautomation system 72 based on the virtual positioning of the secondremote user 82B relative to the virtual positioning of the industrialautomation system 72 in the virtual coordinate system. The processingsystem 62 may determine movement (e.g., physical movement) of the secondremote user 82B in the second remote environment 292, determinecorresponding movement (e.g., virtual movement) of the second remoteuser 82B to an updated virtual positioning in the virtual coordinatesystem, and cause the first extended reality device 52A to presentupdated image data based on the updated virtual positioning of thesecond remote user 82B relative to the virtual positioning of theindustrial automation system 72 in the virtual coordinate system (e.g.,image data that includes a new location of the second remote user 82Bwithin the industrial automation system 72). The processing system 62may also determine movement (e.g., physical movement) of the firstremote user 82A in the first remote environment 292A, determinecorresponding movement (e.g., virtual movement) of the first remote user82A to an updated virtual positioning in the virtual coordinate system,and cause the first extended reality device 52A to present updated imagedata based on the updated virtual positioning of the first remote user82A relative to the virtual positioning of the industrial automationsystem 72 (e.g., image data that includes a new view of the industrialautomation system 72 and the second remote user 82B).

The processing system 62 may further cause the extended reality devices52 to present other output representative data corresponding to inputsensory data associated with the industrial automation system 72 and/orany of the remote users 82 (e.g., based on detections made by thesensor(s) 78 and/or the extended reality devices 52). In certainembodiments, the processing system 62 may determine a respective subsetof the input sensory data that is associated with each of the virtualpositionings of the remote users 82, such as by determining the virtualpositioning of the sensor(s) 78 in the virtual coordinate system andselecting sensor data received from the respective sensor(s) 78 havingvirtual positionings that are within a threshold distance of therespective virtual positionings of the remote users 82. The processingsystem 62 may then determine corresponding output representative dataderived from the respective subset of the input sensory data forpresentation via the extended reality devices 52.

In this manner, from a plurality of received input sensory data (e.g.,associated with an entirety of the industrial automation system 72), theprocessing system 62 may identify a subset of input sensory datarelevant to (e.g., at a portion of the industrial automation system 72proximate to) the virtual positioning of the first remote user 82A, suchas operating data associated with an industrial automation component 74.The processing system 62 may determine movement of the first remote user82A in the first remote environment 292A, such as based on positioningdata received from the first extended reality device 52A (e.g., via amovement sensor of the first extended reality device 52A), and determinecorresponding movement of the first remote user 82A to an updatedvirtual positioning in the virtual coordinate system. In response, fromthe plurality of received input sensory data, the processing system 62may identify another subset of input sensory data relevant to (e.g., atanother portion of the industrial automation system 72 proximate to) theupdated virtual positioning the first remote user 82A, such as operatingdata associated with a different industrial automation component 74proximate to the updated virtual positioning of the first remote user82A. As a result, the processing system 62 may determine outputrepresentative data that may be relevant to the first remote user 82Abased on identified subsets of the input sensory data.

The processing system 62 may similarly cause the second extended realitydevice 52B of the second remote user 82B to present outputrepresentative data based on the virtual positionings of the firstremote user 82A, the second remote user 82B, and the industrialautomation system 72 relative to one another, such as based on differentsubsets of input sensory data identified via the virtual positioning ofthe second remote user 82B relative to the virtual positioning of theindustrial automation system 72. In this way, the processing system 62may cause the extended reality devices 52 to present outputrepresentative data that may simulate real-life conditions and sensoryinformation for the remote users 82 with respect to the industrialautomation system 72, thereby improving the collaboration andinteraction between the remote users 82.

To this end, the processing system 62 may receive an input associatedwith mapping into the virtual coordinate system (e.g., to start atrespective virtual positionings within the virtual coordinate system).In some embodiments, the remote users 82 may map into the virtualcoordinate system via the extended reality devices 52. For instance, theremote users 82 may be mapped into the virtual coordinate system viauser inputs indicative of a selection of respective virtual positioningswithin the virtual coordinate system at which the remote users 82 mayinitiate. Moreover, the industrial automation system 72 may be mappedinto the virtual coordinate system. Additional techniques associatedwith mapping into the virtual coordinate system are described in U.S.patent application Ser. No. 17/796,641, which is hereby incorporated byreference in its entirety for all purposes. Mapping of the remote users82 and the industrial automation system 72 may define respective initialvirtual positionings of the remote users 82 and the industrialautomation system 72 within the virtual coordinate system.

Furthermore, the processing system 62 may receive sensor data indicativeof physical movement of the remote users 82 to determine correspondingupdated virtual positionings of the remote users 82 (e.g., relative toone another, relative to the industrial automation system 72) within thevirtual coordinate system. For instance, each of the extended realitydevices 52 may include a movement sensor, such as an accelerometer, agyroscope, a global positioning system, and the like, which may detectmovement of the extended reality device 52 caused by movement of theremote user 82. The processing system 62 may then determine the virtualpositioning in the virtual coordinate system based on positioning andmovement data received from the movement sensor. For example, theprocessing system 62 may receive positioning data indicative that theuser moved a distance within the first remote environment 292A. Inresponse, the processing system 62 may update the virtual positioning ofthe user within the virtual coordinate system to reflect movement of theuser by the distance. The processing system 62 may then update theoutput representative data presented to the user based on the updatedvirtual positioning of the user.

FIG. 9 is a schematic diagram of an embodiment of a communicationnetwork 310 in which the first remote user 82A, the second remote user82B, and the industrial automation system 72 are mapped into a virtualcoordinate system 312 corresponding to the industrial automation system72. The processing system 62 may reference the virtual coordinate system312 to determine output representative data to be presented. In someembodiments, the virtual coordinate system 312 may be stored in theprocessing system 62 (e.g., in the memory 114 of the processing system62). In additional embodiments, the virtual coordinate system 312 may bestored in a different component or system (e.g., a cloud server)communicatively coupled to the processing system 62, and the processingsystem 62 may access the virtual coordinate system 312 via thecomponent.

In the illustrated embodiment, the first remote user 82A is mapped intoa first virtual positioning 314, the second remote user 82B is mappedinto a second virtual positioning 316, and the industrial automationsystem 72 is mapped into a third virtual positioning 318 in the virtualcoordinate system 312. The processing system 62 may determine outputrepresentative data to be presented based on the virtual positionings314, 316, 318 relative to one another. In certain embodiments, theprocessing system 62 may present the output representative data in aspatial manner based on the virtual positionings 314, 316, 318 tosimulate the physical positioning of the users 82 within the industrialautomation system 72 and relative to one another. For instance, theprocessing system 62 may determine that audio output is being generatedby the industrial automation system 72 at a particular area of (e.g., aparticular location) the third virtual positioning 318. The processingsystem 62 may determine the first virtual positioning 314 relative tothe area of the third virtual positioning 318 and, based on thecomparison, the processing system 62 may cause the first extendedreality device 52A to present corresponding output representative databased on the first virtual positioning 314 relative to the area of thethird virtual positioning 318. By way of example, the third virtualpositioning 318 may be to the right of the first virtual positioning314, and the processing system 62 may therefore cause the first extendedreality device 52A to present output representative data (e.g., audiooutput) in a manner that corresponds to the area of the third virtualpositioning 318 being to the right of the first virtual positioning 314(e.g., audio output is presented at the right side of the first remoteuser 82A). As another example, the third virtual positioning 318 may befarther from the first virtual positioning 314 than to the secondvirtual positioning 316. As such, the processing system 62 may cause thefirst extended reality device 52A to present output representative data(e.g., audio output) at a lesser intensity (e.g., a lower volume) thanthat presented by the second extended reality device 52B.

The processing system 62 may monitor movement of the remote users 82(e.g., within the respective remote environments 292), determinecorresponding movement of the respective virtual positionings 314, 316to updated virtual positionings 314, 316 based on the movement of theremote users 82, and cause the extended reality devices 52 to presentupdated output representative data based on the updated virtualpositionings 314, 316, 318 relative to one another. For example, theprocessing system 62 may cause the extended reality devices 52 to updatethe spatial manner in which the output representative data is presented.Thus, as the remote users 82 move within their respective remoteenvironments 292, the presentation of the output representative dataassociated with the remote users 82 and/or the industrial automationsystem 72 may reflect such movement, thereby simulating real-lifemovement of the remote users 82 relative to one another and/or to theindustrial automation system 72.

The processing system 62 may also cause the extended reality devices 52to present feedback provided by the remote users 82 based on the virtualpositionings 314, 316 between the remote users 82 in the virtualcoordinate system 312. For instance, audio data associated with thesecond remote user 82B (e.g., an audio input provided by the secondremote user 82B and/or audio data associated with the second remoteenvironment 292B) may be received via the second extended reality device52B, the processing system 62 may determine the first virtualpositioning 314 of the first remote user 82A relative to the secondvirtual positioning 316 of the second remote user 82B, and theprocessing system 62 may then cause the first extended reality device52A to provide feedback (e.g., audio feedback) corresponding to theaudio data received via the second extended reality device 52B and basedon the first virtual positioning 314 relative to the second virtualpositioning 316. For instance, the processing system 62 may cause thefirst extended reality device 52 to present corresponding audio data ina spatial manner (e.g., based on a directionality of the second virtualpositioning 316 relative to the first virtual positioning 314) and at anintensity (e.g., based on a distance between the first virtualpositioning 314 relative to the second virtual positioning 316) tosimulate the remote users 82 communicating with one another within thesame physical environment.

In some embodiments, the processing system 62 may also determine thatany of the virtual positionings 314, 316, 318 intersect with one anotherand cause the extended reality devices 52 to present feedback based onthe virtual positionings 314, 316, 318 intersecting with one another.Such feedback may be presented to represent or simulate physical contact(e.g., as spatial feedback). By way of example, the processing system 62may determine a location on the virtual positionings 314, 316, 318 wherethe intersection occurs and cause the extended reality devices 52 topresent the feedback based on the location (e.g., relative to thevirtual positionings 314, 316). In an example embodiment, the firstextended reality device 52A may include multiple components configuredto present feedback (e.g., haptic feedback). The processing system 62may determine a location on the first virtual positioning 314 where thefirst virtual positioning and the third virtual positioning 318intersect with one another. The processing system 62 may then determinea subset of the components corresponding to the location on the firstvirtual positioning and cause the subset of the components to presentfeedback. For instance, based on the location of intersection occurringon a right portion of the first virtual positioning 314, the processingsystem 62 may select a subset of components that are located on theright side of the first remote user 82A (e.g., equipped or worn on theright side of the first remote user 82A) and cause such components topresent feedback.

To this end, the processing system 62 may determine a positioning of thecomponents relative to one another in order to determine the subset ofcomponents corresponding to the location of intersection. In thismanner, the processing system 62 may cause presentation of feedback thatmore closely simulates physical contact represented by the intersectionbetween the virtual positionings 314, 316, 318.

In certain embodiments, a particular industrial automation component 74of the industrial automation system 72 may be mapped into the virtualcoordinate system 312, such as to a fourth virtual positioning 320, as aresult of the industrial automation system 72 being mapped into thevirtual coordinate system 312. In other words, specific parts orequipment of the industrial automation system 72 may be mapped into thevirtual coordinate system 312. The processing system 62 may cause theextended reality devices 52 to present output representative dataspecifically associated with the industrial automation component 74based on the virtual positionings of the remote users 82 correspondingto the fourth virtual positioning 320 of the industrial automationcomponent 74. As an example, in response to a determination that adistance between the first virtual positioning 314 and the fourthvirtual positioning 320 is within a threshold distance, the processingsystem 62 may cause the first extended reality device 52A to presentoutput representative data associated with the industrial automationcomponent 74, such as operating data specific to the industrialautomation component 74. As another example, the processing system 62may determine a field of view of the first remote user 82A associatedwith the first virtual positioning 314 within the virtual coordinatesystem 312 (e.g., based on image data received by the first extendedreality device 52A), and the processing system 62 may cause the firstextended reality device 52A to present output representative dataassociated with the industrial automation component 74 based on thefourth virtual positioning 320 being within the field of view associatedwith the first virtual positioning 314.

In this manner, the processing system 62 may cause additionalpotentially relevant information to be provided to the remote users 82based on the virtual positionings 314, 316 of the remote users 82relative to the industrial automation component 74. Such information maybe helpful for the remote users 82 to perform tasks and interactions,such as a maintenance operation, associated with the industrialautomation component 74. Although the example discussed herein isdirected to presenting output representative data associated with theindustrial automation component 74, output representative dataassociated with other parts of the industrial automation system 72, suchas a particular subsystem, a particular location, a particular section,and the like, of the industrial automation system 72, may be presentedin other embodiments based on the virtual positionings 314, 316 of theremote user 82 relative to the other parts of the industrial automationsystem 72.

In further embodiments, another physical object 322 may be mapped intothe virtual coordinate system 312, such as to a fifth virtualpositioning 324. The physical object 322 is located in the first remoteenvironment 292A in the illustrated environment, but the physical object322 may be located in the second remote environment 292B, the industrialautomation system 72, and/or at any other suitable environment inadditional embodiments. As an example, the physical object 322 mayinclude a wall, a table, or any other object 322 that may be positionedin the first remote environment 292A. The processing system 62 may causeoutputs to be provided based on the fifth virtual positioning 324 of thephysical object 322 relative to any of the other virtual positionings314, 316, 318, 320 to simulate a physical presence of the physicalobject 322. In an example, image data and/or audio data associated withthe physical object 322 may be presented by the second extended realitydevice 52B (e.g., relative to the industrial automation system 72) basedon the fifth virtual positioning 324 of the physical object 322 relativeto the third virtual positioning 318 of the industrial automation system72. In an additional example, feedback that simulates a physicalinteraction (e.g., physical contact) with the physical object 322 may bepresented. For instance, the processing system 62 may determine that thesecond virtual positioning 316 of the second remote user 82B and thefifth virtual positioning 324 of the physical object 322 intersect withone another, and the processing system 62 may therefore cause the secondextended reality device 52B to present feedback (e.g., spatial hapticfeedback based on a location of the intersection) in response.

The presented feedback associated with the physical object 322 mayprovide a realistic interaction between the physical object 322 andanother entity that is not physically located within the first remoteenvironment 292A. For instance, the processing system 62 may cause thefirst extended reality device 52A to present image data including avirtual representation of the second remote user 82B. The intersectionbetween the second virtual positioning 316 of the second remote user 82Band the fifth virtual positioning 324 of the physical object 322 mayindicate an interaction between the virtual representation of the secondremote user 82B and the physical object 322 as provided by the firstextended reality device 52A and as seen by the first remote user 82A. Asa result, the feedback provided to the second remote user 82B maysimulate a physical sensation experienced by the second remote user 82Bwhen interacting with the physical object 322 to simulate the secondremote user 82B being physically located within the first remoteenvironment 292A. For example, the physical object 322 may include awall or boundary of the first remote environment 292A, and the feedbackpresented by the second extended reality device 52B may guide the secondremote user 82B to move away from the physical object 322 such that thephysical object 322 does not block the virtual representation of thesecond remote user 82B presented to the first remote user 82A. Thus, theprocessing system 62 may increase a realistic simulation of the secondremote user 82B being physically within the first remote environment292A.

The physical object 322 may be mapped into the virtual coordinate system312 via a user input in some embodiments. As an example, the firstremote user 82A may utilize the first extended reality device 52A todefine the fifth virtual positioning 324 of the physical object 322. Inadditional embodiments, the physical object 322 may be automaticallymapped into the virtual coordinate system 312, such as based on datareceived from the first extended reality device 52A. The data mayindicate the positioning (e.g., a physical positioning) of the physicalobject 322 relative to the first remote user 82A, and the processingsystem 62 may map the physical object 322 to the fifth virtualpositioning 324 (e.g., relative to the first virtual positioning 314 ofthe first remote user 82A) based on the indicated positioning of thephysical object 322 relative to the first remote user 82A. In anexample, the data may include sensor data received from a proximitysensor, a distance sensor, and/or a ranging sensor. In another example,the data may include image data (e.g., of the first remote environment292A). The processing system 62 may identify the physical object 322 anddetermine the positioning (e.g., a physical positioning) of the physicalobject 322 relative to the first remote user 82A based on the image datato map the physical object 322 into the virtual coordinate sy stem 312.

FIG. 10 is a schematic diagram of an embodiment of a communicationnetwork 350 for enabling multiple user collaboration between the firstremote user 82A located within the first remote environment 292A, thesecond remote user 82B located within the second remote environment292B, and a third remote user 82C located within a third remoteenvironment 292C. Each of the remote users 82 and the industrialautomation system 72 may be mapped into the virtual coordinate system312, and the processing system 62 may cause the extended reality devices52 to present output representative data based on the virtualpositionings in the virtual coordinate system 312. In this manner, eachof the remote users 82 and the industrial automation system 72 may bevirtually positioned within a virtual environment 352 associated withthe virtual coordinate system 312.

In the illustrated embodiment, respective virtual representations 354 ofthe first remote user 82A and the second remote user 82B may bepresented in the virtual environment 352. That is, the processing system62 may cause each of the extended reality devices 52B, 52C of the secondand third remote users 82B, 82C, respectively, to present image datathat includes a first virtual representation 354A of the first remoteuser 82A located within the industrial automation system 72.Additionally, the processing system 62 may cause each of the extendedreality devices 2A, 52C of the first and third remote users 82A, 82C,respectively, to present image data that includes a second virtualrepresentation 354B of the second remote user 82B located within theindustrial automation system 72. For example, the virtualrepresentations 354 of each of the first remote user 82A and the secondremote user 82B may be presented based on the movement of the firstremote user 82A and the second remote user 82B in the remoteenvironments 292A, 292B, respectively. Furthermore, using the techniquesdescribed herein, output representative data may be presented to thefirst and second remote users 82A, 82B based on the virtual positioningsof the remote users 82A, 82B in the virtual coordinate system 312, andfeedback may also be presented to the first and second remote users 82A,82B based on interactions associated with the virtual positionings ofthe remote users 82A, 82B (e.g., based on the virtual positioningsintersecting with one another) in the virtual coordinate system 312.

Output representative data may also be presented to the third remoteuser 82C based on the virtual positioning of the third remote user 82Cin the virtual coordinate system 312. For instance, the processingsystem 62 may cause the third extended reality device 52C to presentoutput representative data associated with the industrial automationsystem 72 (e.g., output representative data associated with operatingdata of the industrial automation component 74) based on the virtualpositioning of the third remote user 82C relative to the industrialautomation system 72. Furthermore, the processing system 62 may causethe third extended reality device 52C to present output representativedata 356 associated with the other remote users 82A, 82B based on thevirtual positioning of the third remote user 82C relative to the virtualpositionings of other remote users 82A, 82B. Such output representativedata 356 may include identifier information (e.g., a name, a job title,contact information, experience level), a task being performed,biometric data (e.g., a stress level), or any combination thereof.

As an example, the processing system 62 may cause the third extendedreality device 52C to present output representative data 356 associatedwith one of the other remote users 82 based on the virtual positioningof the third remote user 82C corresponding to the virtual positioning ofthe other remote user 82 (e.g., based on the virtual positionings beingwithin a threshold distance of one another, based on the virtualpositioning of the other remote user 82 being within a field of viewassociated with the virtual positioning of the third remote user 82C).For instance, the respective output representative data 356A, 356B maybe presented adjacent to the associated virtual representations 354A,354B of the remote users 82A, 82B. That is, first output representativedata 356A of the first remote user 82A may be presented (e.g., as imagedata) adjacent to the first virtual representation 354A, and secondoutput representative data 356B of the second remote user 82B may bepresented adjacent to the second virtual representation 354B. Thus, thethird remote user 82C may be provided with location-based informationregarding both the industrial automation component 74 and other remoteusers 82.

In certain embodiments, output representative data associated with thethird remote user 82C may be presented (e.g., to the first remote user82A via the first extended reality device 52A, to the second remote user82B via the second extended reality device 52B). For instance, a thirdvirtual representation 354C of the third remote user 82C may bepresented based on the virtual positioning of the third remote user 82C(e.g., relative to the virtual positionings of the first and secondremote users 82A, 82B). Thus, each remote user 82 may be able to view avirtual representation 354 of another of the remote users 82.Furthermore, feedback may be presented based on interactions (e.g.,intersections) with the virtual positioning of the third remote user82C.

In additional embodiments, output representative data, such as a virtualrepresentation, associated with the third remote user 82C may not bepresented (e.g., to the first remote user 82A via the first extendedreality device 52A, to the second remote user 82B via the secondextended reality device 52B), and feedback may not be presented based onan interaction associated with the virtual positioning of the thirdremote user 82C (e.g., the virtual positioning of the third remote user82C intersecting with a virtual positioning of another remote user 82and/or of the industrial automation system 72).

That is, feedback may not be presented to the third remote user 82C(e.g., via the third extended reality device 52C) based on interactionsassociated with the virtual positioning of the third remote user 82C inthe virtual coordinate system 312. For example, feedback may not beprovided to either the first remote user 82A or the third remote user82C when the virtual positioning of the third remote user 82C and thevirtual positioning of the first remote user 82A intersect with oneanother. In this way, the third remote user 82C may be a third-partyuser who observes the other remote users 82A, 82B with respect to theindustrial automation system 72 without directly interacting with theother remote users 82A, 82B or the industrial automation system 72.

In further embodiments, the third remote user 82C may be located in oneof the other remote environments 292A, 292B. As an example, the thirdremote user 82C may be physically proximate to one of the other remoteusers 82A, 82B. For this reason, the third remote user 82C (e.g.,located within the first remote environment 292A) may be able tophysically see or observe one of the other remote users 82, such as thefirst remote user 82A, but not another one of the remote users 82, suchas the second remote user 82B. In this example, the processing system 62may cause the third extended reality device 52C to present the secondvirtual representation 354B of the second remote user 82B, because thethird remote user 82C may not be able to physically see the secondremote user 82B. However, the processing system 62 may not cause thethird extended reality device 52C to present the first virtualrepresentation 354A of the first remote user 82A, because the thirdremote user 82C is able to physically see the first remote user 82A. Forinstance, based on an indication (e.g., a user input, location data)that the third remote user 82C is physically within a threshold distanceof another remote user 82, the processing system 62 may not cause theextended reality device 52C to present the virtual representation 354 ofthe other remote user 82 to avoid causing the third remote user 82C tophysically see the other remote user 82 and also see a separate, virtualrepresentation 354 of the same other remote user 82.

In some embodiments, the processing system 62 may determine movement ofthe virtual positioning of the third remote user 82C in the virtualcoordinate system 312 based on movement of the third remote user 82C inthe third remote environment 292C (e.g., as indicated by positioningdata transmitted via the third extended reality device 52C). Inadditional embodiments, the processing system 62 may receive a userinput indicative of movement of the virtual positioning of the thirdremote user 82C in the virtual coordinate system 312 and determinemovement of the virtual positioning of the third remote user 82C basedon the user input. As an example, the user input may indicate aselection of a target location, position, and/or orientation relative tothe industrial automation system 72 and/or an adjustment of a currentlocation, position, and/or orientation relative to the industrialautomation system 72. In further embodiments, the user input may includea selection of one of the other remote users 82. In such embodiments,the processing system 62 may associate the virtual positioning of thethird remote user 82C with the virtual positioning of the selectedremote user 82. That is, movement of the selected remote user 82 (e.g.,within their remote environment 292) may cause corresponding movement ofboth the virtual positioning of the first remote user 82A and thevirtual positioning of the third remote user 82C. As a result, theprocessing system may cause the third extended reality device 52C of thethird remote user 82C to simulate the experience provided to theselected remote user 82. That is, the third remote user 82C may undergothe same or similar experience as that of the selected remote user 82.For example, the processing system 62 may also cause the third extendedreality device 52C of the third remote user 82C to present the same orsimilar output representative data presented to the selected remote user82 via their extended reality device 52, such as based on movement andpositioning of the selected remote user 82 instead of based on themovement and positioning of the third remote user 82C.

FIG. 11 is a flowchart of a method or process 380 for presenting outputrepresentative data based on virtual positionings in the virtualcoordinate system 312. At block 382, the processing system 62 mayreceive input sensory data, such as from the sensor(s) 78, which mayinclude a sensor disposed in the industrial automation system 72. Atblock 384, the processing system 62 may determine a virtual positioningof a user within the virtual coordinate system 312, which corresponds tothe industrial automation system 72. As an example, the user mayinitially map into a starting or initial virtual positioning in thevirtual coordinate system 312 using the extended reality device 52. Theprocessing system 62 may also determine adjustment of the virtualpositioning within the virtual coordinate system 312. In someembodiments, the processing system 62 may determine adjustment of thevirtual positioning based on positioning or movement data received fromthe extended reality device 52. That is, the processing system 62 maydetermine physical movement of the extended reality device 52 anddetermine corresponding virtual movement of the virtual positioning inthe virtual coordinate system 312. In additional embodiments, theprocessing system 62 may receive a user input indicative of theadjustment of the virtual positioning. The processing system 62 maytherefore determine adjustment of the virtual positioning based on theuser input.

At block 386, the processing system 62 may determine the virtualpositioning of the user relative to the virtual positionings associatedwith other entities mapped into the virtual coordinate system 312, suchas by comparing multiple virtual positionings to one another within thecontext of the virtual coordinate system 312. The other entities mayinclude, for example, the industrial automation system 72 (e.g., anindustrial automation component 74), another user, a physical object,and so forth. At block 388, the processing system 62 may determineoutput information to be presented based on the received input sensorydata and in accordance with the virtual positioning of the user relativeto the other virtual positionings associated with the other entities.Thus, the presented output information may be location-based. In someembodiments, the output information may include output representativedata that is derived based on the input sensory data (e.g., a subset ofthe input sensory data proximate to the virtual positioning of theuser), such as operating data associated with the industrial automationsystem 72.

Such output information may be presented to simulate the user beingphysically positioned within the industrial automation system 72 and maybe indicative of an environment of the industrial automation system 72,an operating parameter of the industrial automation system 72, and thelike. In additional embodiments, the output information may includeoutput representative data associated with other users mapped into thevirtual coordinate system 312. For instance, the information may bespecifically related to another user (e.g., to help identify the otheruser). In further embodiments, the output information may includefeedback indicative of an interaction associated with the user. As anexample, the feedback may be provided based on the virtual positioningof the user and the virtual positioning of another entity (e.g., anotheruser, an industrial automation component 74) intersecting with oneanother. As another example, the feedback may be provided based on audiodata associated with a different user mapped into the virtual coordinatesystem 312, such as an audio input provided by the other user and/oraudio data received from an environment in which the other user islocated. Thus, the feedback may be provided to simulate a real-worldinteraction made between users. At block 390, the processing system 62may then cause the extended reality device 52 to present the outputinformation, thereby providing information to a user based on theirvirtual positioning compared to other virtual positionings (e.g., ofother users, of the industrial automation component 74) within thevirtual coordinate system 312.

Remote Machine Control

In certain embodiments, a user may control operation of a machine withinthe industrial automation system 72. For instance, the user may controlmovement of the machine may within the industrial automation system 72.As an example, the machine may include a robot or other mechanicalsystem, and the user may control movement of the machine to performvarious actions or tasks with respect to the industrial automationsystem 72. It may also be desirable to provide different outputrepresentative data to facilitate the user controlling the machine, suchas to guide the user to control the machine in a desirable manner.

FIG. 12 is a schematic diagram of an embodiment of a communicationnetwork 410 in which a user 412 may control operation of a machine 414that is located within the industrial automation system 72. The machine414 may perform one or more mechanical operations, such as to rotateand/or translate. Indeed, the machine 414 may include various components(e.g., limbs) that are movable relative to one another, such as to movethe machine 414 between locations within the industrial automationsystem 72, to adjust orientations, to interact with an industrialautomation component 74, and so forth. As an example, the machine 414may include a robot, electro-mechanical equipment, and the like. Theuser 412 may control the machine 414 from a different location relativeto the location of the machine 414. For instance, the user 412 may be aremote user 82 located remotely relative to the industrial automationsystem 72 and the machine 414. Thus, the remote user 82 may control themachine 414 from a remote environment. Additionally, the user 412 may bea local user 70 located within the industrial automation system 72, andthe user 412 and the machine 414 may be at different locations withinthe industrial automation system 72.

Movement of the machine 414 may be enabled via actuators (e.g., a linearactuator, rotary actuator, motor). The user 412 may transmit a userinput that is indicative of a request to move the machine 414. The userinput may cause activation of the actuators and subsequent movement ofthe components to perform various actions and operations, such as tochange orientation with respect to the industrial automation system 72,to navigate through different locations within the industrial automationsystem 72, to contact an industrial automation component 74 of theindustrial automation system 72, and the like. In certain embodiments,the processing system 62 may be used to instruct the actuators to movethe machine 414. By way of example, the user 412 may utilize theextended reality device 52 to control the machine 414. Indeed, the user412 may transmit a user input to the extended reality device 52 toindicate a request to move the machine 414, the processing system 62 mayreceive the user input via the extended reality device 52, and theprocessing system 62 may send a command to the machine 414 based on theuser input to move the machine 414 via the actuators. In additionalembodiments, the user 412 may control the machine 414 in a differentmanner, such as by utilizing a processing system separate from theprocessing system 62 in order to move the machine 414.

In an example embodiments, the processing system 62 may determinemovement, such as a gesture, made by the user 412 and causecorresponding movement of the machine 414. For instance, the processingsystem 62 may identify a change in location, orientation, and/or pose ofthe user 412, and cause a corresponding change in location, orientation,and/or pose of the machine 414. Thus, the processing system 62 may causethe machine 414 to mimic movement of the user 412. By way of example,the processing system 62 may determine that the user 412 has physicallymoved a first distance in a first direction, and the processing system62 may cause the machine 414 to move a second distance corresponding tothe first distance and in a second direction corresponding to the firstdirection. Indeed, the user 412 may utilize a suitable amount of spaceavailable within the physical environment of the user 412 (e.g., insteadof machinery confined to a particular physical location and havingmechanical movement limits) to control the machine 414, thereby enablingthe user 412 to control the machine 414 more desirably and/or acutely.

The processing system 62 may cause the extended reality device 52 topresent output representative data based on the positioning (e.g.,position, location, orientation) of the machine 414 within theindustrial automation system 72. In some embodiments, the processingsystem 62 may determine output representative data based on inputsensory data associated with the industrial automation system 72 (e.g.,received via the sensor(s) 78). For example, the processing system 62may determine the positioning of the machine 414 within the industrialautomation system 72, identify a subset of input sensory data associatedwith a portion of the industrial automation system 72 proximate to thepositioning of the machine 414 (e.g., input sensory data associated withan industrial automation component 74 proximate to the machine 414),determine output representative data corresponding to the subset of theinput sensory data, and transmit the output representative data to theextended reality device 52 for presentation. In this manner, theprocessing system 62 may provide the user 412 with informationcorresponding with data that may be available at the positioning of themachine 414 in order to simulate the user 412 being physically at thepositioning of the machine 414.

Indeed, the output representative data, such as image data representinga viewing perspective of the machine 414 at the positioning within theindustrial automation system 72, image data that includes avisualization of the machine 414 within the industrial automation system72, audio data that includes sounds that are audible at the positioningwithin the industrial automation system 72, information regarding anoperating status or operating condition associated with the industrialautomation system 72 (e.g., the industrial automation component 74) atthe positioning of the machine 414 within the industrial automationsystem 72, and the like, may guide the user 412 to determine control ofthe machine 414. The output representative data may, for instance, guidethe user 412 to control the machine 414 to perform a particular actionor task (e.g., my mimicking the movement of the user 412) with respectto the industrial automation system 72, such as to inspect theindustrial automation component 74. As an example, the processing system62 may cause the extended reality device 52 to present a virtualrepresentation of a physical object (e.g., the industrial automationcomponent 74) that is proximate to the machine 414. The user 412 mayperform a movement with respect to the virtual representation of thephysical object (e.g., virtually grasping the industrial automationcomponent 74). Based on the movement performed by the user 412, theprocessing system 62 may cause the machine 414 to perform acorresponding movement (e.g., physically grasping the industrialautomation component 74) that may facilitate performing a task desiredby the user 412.

In additional embodiments, the output representative data may be basedon input sensory data associated with the machine 414, such as inputsensory data received via one or more sensors 416 disposed on themachine 414. As an example, the information may be indicative of anoperating parameter associated with an environment of the industrialautomation system 72, such as a temperature, humidity, atmosphericpressure, wind speed, and so forth, as detected by the sensor(s) 416.Presentation of such output representative data may help simulate theuser being at the positioning of the machine 414 within the industrialautomation system 72. For instance, the sensor(s) 78 and/or thesensor(s) 416 may include a temperature sensor that determines atemperature within the industrial automation system 72. The processingsystem 62 may cause the extended reality device 52 to output hapticfeedback based on data received from the temperature sensor and inaccordance with the positioning of the machine 414 within the industrialautomation system 72. The haptic feedback may, as an example, includeheat that is at a temperature level corresponding to a temperature valuedetermined by the temperature sensor. That is, the processing system 62may cause the extended reality device 52 to output heat at a firsttemperature level based on a first determined temperature and to outputheat at a second temperature level based on a second determinedtemperature, the second temperature level being greater than the firsttemperature level and the second determined temperature being greaterthan the first determined temperature. Additionally, the processingsystem 62 may cause the extended reality device 52 to output otherfeedback based on the temperature value determined by the temperaturesensor, such as a different type of haptic feedback (e.g., a vibration),a visual output (e.g., text indicating the measured temperature), anaudio output (e.g., a sound indicating the measured temperature), and soforth. The processing system 62 may similarly cause the extended realitydevice 52 to present other output representative data (e.g., image data)based on other operating parameters associated with the environment ofthe industrial automation system 72 to provide environmental conditionssensed by the machine 414 in the industrial automation system 72.

Moreover, the processing system 62 may cause the extended reality device52 to present additional feedback based on a determination that one ofthe operating parameters (e.g., a temperature value) associated with theenvironment of the industrial automation system 72 is outside of a rangeof predetermined values (e.g., a range of temperature values). By way ofexample, the operating parameter being outside of the range ofpredetermined values may indicate that the environmental condition maybe imparting an undesirable amount of stress (e.g., high amount of heat)onto the machine 414. For instance, in response to determining thetemperature value exceeds the range of temperature values, theprocessing system 62 may cause the extended reality device 52 to presentvibration in addition to heat output. Thus, the additional feedback mayguide the user 412 to control the machine 414 to navigate to a differentlocation in the industrial automation system 72 where the environmentalcondition does not impart the undesirable amount of stress onto themachine 414.

As a further example, the processing system 62 may cause the extendedreality device 52 to present feedback associated with active movement ormotion of the machine 414. For instance, the output representative datamay be indicative of activation of the actuators and/or movement of thecomponents of the machine 414 relative to one another. Such outputrepresentative data may provide the user 412 with simulation of movementwithin the industrial automation system 72 and further simulatepositioning within the industrial automation system 72. Such outputrepresentative data may include haptic feedback, such as vibration,heat, applied pressure, for example. In some embodiments, the extendedreality device 52 may include a first plurality of components that mayeach provide haptic feedback. Each component of the first plurality ofcomponents may be associated with a corresponding component of thesecond plurality of components of the machine 414. In such embodiments,the processing system 62 may determine movement of a first component ofthe second plurality of components of the machine 414, identify asecond, associated component of the first plurality of components of theextended reality device 52 based on the first component, and instructthe extended reality device 52 to cause the second component to providehaptic feedback. By way of example, the first component may correspondto an arm of the machine 414 (e.g., an arm of a robot), and the secondcomponent may be located at an arm of the user 412. However, the firstplurality of components may provide haptic feedback to any suitable bodypart, such as the face (e.g., the nose, mouth), a leg, a hand, thetorso, and so forth, of the user 412. In additional embodiments, thefirst plurality of components may also provide any other suitablelocation-based feedback, such as a visual and/or audio output, inaccordance with the movement of the second plurality of components ofthe machine 414. In this manner, the processing system 62 may causefeedback to be selectively provided and better simulate movement of themachine 414.

The feedback associated with active movement or motion of the machine414 may also indicate to the user 412 whether the movement of themachine 414 is desirable. As an example, the processing system 62 maydetermine a value associated with movement of the machine 414 (e.g.,movement of the plurality of components of the machine 414), such as aforce imparted onto the machine 414, a torque output by the machine 414,an angle formed between certain components of the machine 414, amovement speed of a component of the machine 414, or any combinationthereof. In response to determining the value is above a thresholdmovement value, which may indicate that the movement of the machine 414is imparting an undesirable amount of stress onto the machine 414, theprocessing system 62 may instruct the extended reality device 52 topresent feedback, such as a notification, to the user 412. The feedbackmay cause the user 412 to adjust movement of the machine 414 (e.g., suchthat the value is below the threshold movement value) to avoid impartingadditional stress onto the machine 414. Additionally, the processingsystem 62 may send a command to cause movement of the machine 414 inresponse to determining the value is above the threshold movement value.That is, the processing system 62 may automatically cause movement ofthe machine 414 to avoid excessive stress from being imparted onto themachine 414. In an example, a user input may cause first movement of themachine 414 that results in the value being above the threshold movementvalue, and the processing system 62 may cause second movement of themachine 414 that overrides the first movement of the machine to reducethe value below the threshold movement value.

The processing system 62 may further cause the extended reality device52 to provide other information, such as feedback regarding actionsperformed by the machine 414. For example, the processing system 62 maymonitor an interaction between the machine 414 and the industrialautomation system 72, such as a contact between the machine 414 and anindustrial automation component 74, and provide feedback based on theinteraction. The feedback may, for instance, be indicative of whetherthe interaction is desirable (e.g., whether the interaction matches atarget interaction) and/or may guide the user 412 to control the machine414 to perform a desirable interaction. Thus, the processing system 62may transmit the feedback to facilitate the user 412 controlling themachine 414.

FIG. 13 is a flowchart of a method or process 440 for presenting outputrepresentative data based on a positioning of the machine 414 in theindustrial automation system 72. At block 442, the processing system 62may determine the positioning of the machine 414 in the industrialautomation system 72. In some embodiments, the processing system 62 maydetermine the positioning of the machine 414 based on sensor datareceived from the sensor(s) 416 of the machine 414 and/or the sensor(s)78 of the industrial automation system 72, such as a position sensor, anaccelerometer, an imaging sensor, a distance sensor, and the like. Inadditional embodiments, the processing system 62 may determine thepositioning of the machine 414 via the extended reality device 52, suchas via a user input by the user 412. In further embodiments, theprocessing system 62 may determine the positioning of the machine 414with reference to a virtual coordinate system (e.g., the virtualcoordinate system 312). For instance, the processing system 62 maydetermine a virtual positioning of the machine 414 relative to a virtualpositioning of the industrial automation system 72 based on detectedmovement of the machine 414.

At block 444, the processing system 62 may receive input sensory dataassociated with the industrial automation system 72 and/or the machine414. That is, such input sensory data may be received via the sensor(s)78 and/or the sensor(s) 416. As an example, the input sensory data maybe indicative of operating information associated with the industrialautomation system 72, an environment of the industrial automation system72, movement data associated with the machine 414, and the like.

At block 446, the processing system 62 may cause the extended realitydevice 52 to present output representative data based on the receivedinput sensory data and in accordance with the positioning of the machine414 in the industrial automation system 72. The output representativedata may help the user 412 determine an appropriate manner in which themachine 414 is to be controlled, such as to transmit a user input (e.g.,via the extended reality device 52) that adjusts a movement of themachine 414.

In certain embodiments, the output representative data may be presentedto simulate information observable at the positioning of the machine414. As an example, the processing system 62 may identify a subset ofthe received input sensory data corresponding to the positioning of themachine 414 (e.g., using sensor data received from a subset of thesensor(s) 78 that are within a threshold distance of the positioning ofthe machine 414). The processing system 62 may cause the extendedreality device 52 to present output representative data based on thesubset of the received input sensory data. In response to a determinedupdate to the positioning of the machine 414, the processing system 62may identify a different subset of the received input sensory data andcause the extended reality device 52 to present updated outputrepresentative data based on the different subset of the received inputsensory data. In an example, the processing system 62 may present outputrepresentative data that includes information associated with anindustrial automation component 74 having a positioning corresponding tothe positioning of the machine 414 (e.g., the industrial automationcomponent 74 is within a threshold distance of the machine 414, theindustrial automation component 74 is within a field of view of themachine 414). In another example, the processing system 62 may presentoutput representative data that includes information corresponding to anenvironment associated with the positioning of the machine 414. Theoutput representative data provided to the user 412 may enable the user412 to determine appropriate control of the machine 414, such asmovement control within the industrial automation system 72.

Furthermore, the processing system 62 may be configured to compare theinput sensory data to respective threshold values and/or ranges ofvalues and cause the extended reality device 52 to present the outputrepresentative data based on the comparison. As an example, in responseto a determination that an operating parameter value corresponding to anenvironmental condition (e.g., a temperature) associated with thepositioning of the machine 414 is outside of a range of values, theprocessing system 62 may cause the extended reality device 52 to outputa notification to move or re-locate the machine 414 to a differentlocation in the industrial automation system 72. As another example, inresponse to a determination that operating data or condition (e.g.,vibration, temperature) of a nearby industrial automation component 74is outside of a threshold range of values, the processing system 62 maycause the extended reality device 52 to output a notification regardingthe industrial automation component 74, such as that a maintenanceoperation (e.g., an inspection, a replacement) is to be performed on theindustrial automation component 74.

In additional embodiments, the output representative data may bepresented to indicate an active movement or motion of the machine 414.As an example, the output representative data may simulate movement ofthe machine 414 to the user 412, such as by providing spatial hapticfeedback to the user 412 corresponding to detected movement ofcomponents of the machine 414. Thus, the processing system 62 mayprovide the user 412 with a more interactive or realistic experiencewhen controlling the machine 414. As another example, the processingsystem 62 may present the output representative data to block movementthat would cause excessive stress to be imparted onto the machine 414(e.g., on an actuator, on a motor). For instance, the input sensory datamay be indicative of a movement value of the machine 414 (e.g., an angleof rotation, a torque output, a force output, an imparted force, a speedof movement) and, in response to determining that the movement value isgreater than a threshold movement value, the processing system 62 maypresent the output representative data to guide the user to adjustmovement of the machine 414, such as to guide the user 412 adjust themovement of the machine 414. As such, the processing system 62 mayoutput the representative data to block excessive stress from beingimparted onto the machine 414.

FIG. 14 is a flowchart of a method or process for presenting feedbackbased on an operation being performed by the machine 414 with respect tothe industrial automation system 72. At block 472, the processing system62 may determine the operation being performed by the machine 414. Theoperation may, for example, include an interaction between the machine414 and the industrial automation component 74 and may be selected froma plurality of possible operations that the machine 414 may perform. Forinstance, the processing system 62 may determine the operation beingperformed by the machine 414 via a user input. Additionally, theprocessing system 62 may determine the operation (e.g., a maintenanceoperation, a relocation operation) being performed by the machine 414(e.g., by selecting the operation from the plurality of possibleoperations) based on data received from the sensor(s) 78 (e.g., thesensor(s) 78 disposed on the industrial automation component 74) and/orthe sensor(s) 416, such as data indicative of a status (e.g., anoperating condition, diagnostic information) of the industrialautomation component 74 along with data indicative that the location ofthe machine 414 is within a threshold distance of the industrialautomation component 74, the industrial automation component 74 iswithin a field of view of the machine 414, or that a first location ofthe machine 414 corresponds to a second location of the industrialautomation component 74. In some embodiments, a user input, such as arequest to move the machine 414, may control the machine 414 to performthe operation.

At block 474, the processing system 62 may cause the extended realitydevice 52 to present output representative data based on the operationusing any of the techniques described herein. Such output representativedata may correspond to input sensory associated with the positioning ofthe machine 414 in the industrial automation system 72 and/or associatedwith the industrial automation component 74. Indeed, in response todetermining the operation includes an interaction between the machine414 and the industrial automation component 74, the processing system 62may identify information associated with the industrial automationcomponent 74 and transmit the information to the extended reality device52 for presentation. The presented output representative data mayfacilitate control of the machine 414 via user input to perform theoperation.

At block 476, the processing system 62 may determine an operatingparameter value associated with the control of the machine 414. Theoperating parameter value may be indicative of the movement of themachine 414 as controlled by a user input. For instance, the operatingparameter value may include a force imparted onto the machine 414, aforce imparted by the machine 414, a torque output by the machine 414,an angle formed between components of the machine 414, a movement speedof a component of the machine 414, another suitable operating parametervalue, or any combination thereof.

At block 478, the processing system 62 may determine a range of valuesassociated with the operation. The range of values may be indicative ofa desirable performance of the operation. By way of example, eachoperation of the plurality of possible operations may be associated witha respective range of values, and the processing system 62 may determinethe range of values associated with the determined operation. Theprocessing system 62 may compare the operating parameter valueassociated with the control of the machine and the range of valuesassociated with the operation to one another to determine whether thecontrol of the machine 414 is causing the operation to be performed asdesired.

At block 480, the processing system 62 may cause the extended realitydevice 52 to provide feedback in response to a determination that theoperating parameter value is outside of the range of values. In certainembodiments, the feedback may include visual instructions, audioinstructions, haptic feedback, or other suitable information to indicatethat the control of the machine 414 is to be adjusted (e.g., such thatthe operating parameter value associated with the interaction is withinthe threshold range of values). Thus, the processing system 62 may guidethe user 412 to control the machine 414 more desirably, such as insubsequent operations. In additional embodiments, the processing system62 may automatically adjust the machine 414. That is, the processingsystem 62 may override the control caused by a user input in response todetermining the operating parameter value is outside of the range ofvalues, such as to avoid excessive stress from being imparted onto themachine 414 and/or the industrial automation component 74. For example,the processing system 62 may send a command to the machine 414 to blockmovement caused by the user input in response to determining theoperating parameter value is outside of the range of values.

In one example, the processing system 62 may determine that theoperation includes a force being applied by the machine 414 to contactthe industrial automation component 74 (e.g., to actuate a component ofthe industrial automation component 74), such as to press a button, movea switch, turn a dial, and so forth. The processing system 62 may causethe extended reality device 52 to present output representative dataregarding the industrial automation system 72, the industrial automationcomponent 74, and/or the operation in response to determination of theoperation. The processing system 62 may also cause the extended realitydevice 52 to present feedback based on the force being applied by themachine 414. Such feedback may include, for instance, haptic feedback(e.g., a vibration) to simulate a user performing the operation. Indeed,the processing system may adjust the feedback based on the amount offorce being applied, such as by providing a feedback of a higherintensity for a higher amount of force being applied.

Additionally, there may be a range of force values associated with theoperation, such as to enable an adequate amount of force to perform theoperation. Thus, the processing system 62 may determine whether theforce being applied by the machine 414 is within the range of forcevalues. In response to a determination that the force is below the rangeof force values (e.g., indicative that the component of the industrialautomation component 74 is not being adequately actuated by the forceapplied by the machine 414), the processing system 62 may cause theextended reality device 52 to present feedback (e.g., instructions) thatinforms the user 412 to control the machine 414 to impart a greaterforce. Additionally, the processing system 62 may directly cause themachine 414 to impart a greater force. In response to a determinationthat the force is above the range of threshold force values (e.g.,indicative that an undesirable amount of stress may be imparted onto thecomponent by the machine 414), the processing system 62 may cause theextended reality device 52 to present feedback that informs the user 412to control the machine 414 to impart a reduced force. Additionally, theprocessing system 62 may directly cause the machine 414 to impart areduced force. In this manner, the processing system 62 may facilitateoperation of the machine 414 to apply a desirable or target force.

In another example, the processing system 62 may determine that theoperation includes movement of the industrial automation component 74caused by the machine 414, such as to push, pull, turn, rotate, carry,and so forth, the industrial automation component 74. In this example,the processing system 62 may determine a current positioning of theindustrial automation component 74. Furthermore, there may be a range ofpositioning values (e.g., an angle of rotation, a location coordinate)associated with the operation. The processing system 62 may compare thepositioning of the industrial automation component 74 with the range ofpositioning values, and the processing system 62 may provide feedback toguide movement of industrial automation component 74 such that thepositioning is within the range of positioning values. For example, theprocessing system 62 may output visual instructions, audio data (e.g.,spatial audio feedback), haptic feedback (e.g., spatial vibrations)indicative of the positioning relative to the range of positioningvalues. Thus, the feedback may facilitate operation to move theindustrial automation component 74 in a desirable manner via the machine414.

Time-Based User Experiences

Returning to FIG. 1 , the processing system 62 may enable users to betrained or other user experiences to be provided using collected data(e.g., input data 76, output data, 80, output data 84, input data 88).Indeed, as discussed below, user experiences may be generated based onor using the collected data to provide the user an immersive experiencethat relates to the past, the present (e.g., real-time or substantiallyreal-time), or the future conditions of a monitored environment. Thatis, based on the collected data, the processing system 62 may generateoutput representative data (as discussed above) that corresponds to aparticular period of time that occurred in the past, that is presentlyoccurring, or is that predicted to occur in the future. The userexperiences may be provided to local or remote users, for example, totrain the users (e.g., based on previous data), to manage currentoperations (e.g., based on current data), or predict a state of themonitored equipment (e.g., based on an analysis of the past and currentdata). During the user experiences, output representative data (asdiscussed above) may be provided to one or more electronic devicesutilized by the user, such that the user may physically experience theconditions that are present at the industrial automation system 72.

Before discussing the user experiences that may be facilitated by theprocessing system 62, it should be noted that data received by theprocessing system 62 (e.g., input data 76, output data, 80, output data84, input data 88) may be stored in the database 68. Additionally, theprocessing system 62 (e.g., via processing circuitry 112 of FIG. 2 ) mayexecute computer-executable instructions to perform the techniquesdescribed herein. For example, a user such as the local user 70 orremote user 82 may access or interact with user experiences that theprocessing system 62 provides. More specifically, the processing system62 may execute computer-readable instructions to run a computer program(e.g., software) that the local user 70 and remote user 82 may interactwith, and the processing system 62 may provide user experiences based onuser inputs made by the local user 70 and remote user 82. For instance,a user may provide a user input to browse a library of available userexperiences (e.g., particular past experiences, present experiences, andfuture experiences) and to select a particular experience. In response,the processing system 62 may provide the selected user experience to theuser, thereby providing a fully immersive experience that may enable theuser to experience events that have occurred, are currently occurring,or that could occur in the future.

Bearing this in mind, FIG. 15 is a flowchart of a method 500 forproviding a time-based user experience. The method 500 may be performedby the processing system 62 by executing computer-readable instructionsthat may be stored on a computer-readable medium such as non-transitorycomputer-readable medium. Although the method 500 is described as beingperformed in a particular order, it should be noted that the method 500may be performed in any suitable order. Moreover, although the method500 is described as being performed by the processing system 62, themethod 500 may be performed by any suitable computing device.

At block 502, the processing system 62 may provide data regardingavailable user experiences to one or more electronic devices utilized bya user such as, but not limited to, the extended reality device 52 or anelectronic device that is communicatively coupled to the extendedreality device 52. More specifically, the processing system 62 maycategorize or otherwise arrange collected data (e.g., previouslyobtained input data 76, output data, 80, output data 84, and input data88 or current input data 76, output data, 80, output data 84, or inputdata 88) into collections or particular user experiences. For example,as described above, the processing system 62 may store the input data(e.g., received input data, categorized input data) and/or correspondingoutput data in the database 68, and the processing system 62 mayassociate the input data and/or the output data with an identifier, suchas an event (e.g., a maintenance event).

The processing system 62 may provide data regarding available userexperiences by indicating events for which a tag or identifier exists.For example, the user experiences may indicate past events and eventscurrently taking place (or that have very recently occurred). Morespecifically, past events may include a particular occurrence (e.g., amaintenance operation, repair operation, installation operation,inspection operation, or other task performed by someone in anautomation system (e.g., local user 70) or a remote user (e.g., remoteuser 82). Present events may include a task or activity for which datais currently being provided to the processing system 62 or database 68(e.g., data received from an extended reality device 52 utilized by thelocal user 70, the remote user 82, or both). Present events may bedefined with respect to a particular user. The user experiences may alsoindicate one or more future events that, as described below, may begenerated based on collected data. In response to receiving the dataregarding available user experiences, the extended reality device 52 oran electronic device communicatively coupled to the extended realitydevice 52 may display a graphical user interface that visually (orotherwise) indicates the user experiences available to the user. Forinstance, in addition to a graphical user interface, the extendedreality device 52 may convey the user experiences available to the useraudibly. Moreover, the graphical user interface provided may be providedas an overlay in front of a real world setting (e.g., in the case ofproviding augmented reality content). It should also be noted that, insome embodiments, the processing system 62 may not perform theoperations associated with block 502, block 504 (discussed below), orboth in response to determining the user's intent. For example, theprocessing system 62 may determine an experience the user may want touse based on information about the user (e.g., user profile information)and the user's past actions. In such a case, the processing system 62may provide output representative data associated with the experience tothe extended reality device 52 in lieu of providing data regardingavailable user experiences, receiving user input indicative of aselection of a user experience, or both.

In some embodiments, the data regarding available user experiences maybe provided based on input received by the processing system 62. Thatis, the processing system 62 may receive inputs from a menu system or asearch field that allows a user to navigate through the available userexperiences. In some embodiments, the processing system 62 may groupexperiences based on whether the experience is related to a past event,a current operation status, or a future expected state. After receivingthe input to specify a type of user experience (e.g., past, present,future), the processing system 62 may present a list of available userexperiences that have been previously captured, can be generated in realtime based on contemporaneously acquired sensor data, or predicted forfuture states based on collected data and identified trends.

At block 504, the processing system 62 may receive a user inputindicative of a selection of a particular user experience of the listeduser experiences. For instance, the user may make such an input usingthe extended reality device 52, an electronic device communicativelycoupled to the extended reality device 52, or a combination thereof toselect a user experience included in the graphical user interfacementioned above. It should be noted that the user input may also beprovided verbally. For example, a user may verbally request certaincontent, and in response to receiving such a request, the processingsystem 62 may determine output representative data (as discussed below)indicated by the request. Furthermore, it should be noted that, in someembodiments, the user input may also include a user intent (e.g., animplied input) that the processing system 62 may derive from contextualinformation for the user. For example, in response to detecting the usercomplain about, or a noise indicative of the user's dissatisfaction, theprocessing system 62 may determine a user experience (or outputrepresentative data associated with an experience) to address the user'scomplaint or dissatisfaction.

At block 506, the processing system 62 may determine outputrepresentative data to be presented on the extended reality device 52(or an electronic device communicatively coupled to the extended realitydevice 52 or both) based on the user selection of a user experience. Inother words, the processing system 62 may determine the user experienceindicated by the user input and determine output representative data tobe provided to enable the user to experience the selected userexperience. For instance, as noted above, the processing system 62 mayassociate the input data and/or the output data with an identifier, suchas an event (e.g., a maintenance event). Based on a user inputindicative of a request to select a user experience that is associatedwith the identifier, such as to simulate the event, the processingsystem 62 may retrieve the input data and/or the output data associatedwith the identifier to cause the user to be experience a user experience(e.g., via the output representative data being presented on theextended reality device 52).

For example, in the case of a user experience dealing with a past event,output representative data may include information included in the inputsensory datasets previously received and stored in the database 68. Theprocessing system 62 may determine a quantity or amount of suchinformation to be presented via the output representative data. That is,the processing system 62 may determine which information (e.g.,information associated with the user, information associated with theindustrial automation system 72) is to be presented. The processingsystem 62 may additionally determine a format in which the outputrepresentative data is to be presented. For instance, the processingsystem 62 may determine whether different information or data is to bepresented via image data, text, audio data, haptic feedback, anothersuitable format, or any combination thereof. For example, the processingsystem 62 may recreate a previously occurring event such that the usermay experience the same (or a nearly identical) experience in anaugmented or virtual reality setting as experienced by a user associatedwith the input data (e.g., input data 76) that serves as the basis forthe user experience. In other words, a user may “relive” an event as theevent occurred via image data, text (which could characterize orindicate any form of data received by the sensor(s) 78), audio data,haptic feedback that will be provided to the extended reality device 52.As such, the processing system 62 may generate a user experience inwhich a user experiences an event as the event as indicated by dataoriginally collected during that event. In this way, the processingsystem 62 may present a more immersive environment to simulate differentsituations associated with the industrial automation system 72 withouthaving to change a real-world status of the industrial automation system72.

As another example, the output representative data may includeinformation originally sent as output representative data to a differentuser during the original event's occurrence. For example, during a userexperience in which a user experienced a past event, the processingsystem 62 may identify output data that was originally provided toanother user other than the user who originally experienced the event.For instance, in the case in which a user is to observe an event thatoccurred in the industrial automation system 72 from the lens orviewpoint of how the remote user 82 experienced the event, theprocessing system 62 may determine the output representative data bydetermining the output data 84 previously provided to the remote user82, generating output representative data based on the output data 84,or both. Furthermore, in some embodiments, the output representativedata determined at block 506 may include the input data 88 originallygenerated by equipment of the remote user 82 or data generated based offof the input data 88 or sensor data collected by the sensor(s) 78.

Similarly, for a user experience that is generated based on the present(e.g., a presently occurring event or operation involving a local useror remote user), the processing system 62 may identify data receivedfrom an extended reality device 52 of the local user 70, one or moreremote users 82, or both. The processing system 62 may also identify thefirst output data 80, the output data 84 provided to the extendedreality device 52 of the local user 70, remote user(s) 82, or both. Theidentified data (e.g., whether input data, output data, or somecombination thereof) may then be included in the output representativedata that is determined at block 506. Such output representative datamay also include data that is determined based on data collected by thesensor(s) 78. As such, a user may experience a presently occurring eventas the event occurs in real-time or in near real-time as though the userwere experiencing the event firsthand.

For future user experiences occurring in the future, the processingsystem 62 may determine output representative data based on previouslycollected data (e.g., input data received from extended reality devices52 and the sensor(s) 78). The determined output representative data maybe utilized, for example, to provide the user a simulation of eventsthat could occur (e.g., in the industrial automation system 72) from theperspective of a local user or a remote user in light of the currentconditions of the system, current equipment employed by the system, theprevious wear that the equipment has already experienced, the expectedstress and operational use of the equipment in the future, and the like.More particularly, returning briefly to FIG. 1 , the processing system62 (or the processing circuitry 112) may perform machine learning (e.g.,based on a training model), perform simulations, or both utilizing datacollected in the database 68 (e.g., input data 76, output data, 80,output data 84, input data 88) to determine output representative datathat will be used to provide a future, or predictive, user experience.For example, the processing system 62 may execute computer-readableinstructions to perform machine-learning, simulate the operation of oneor more components of the industrial automation system 72, or both todetermine the output representative data. Thus, the user may be placed(e.g., using virtual reality or augmented reality) within a simulationof the industrial automation system 72 that the processing system 62generates based on previously collected data.

Returning to FIG. 15 and the discussion of the method 500, at block 508,the processing system 62 may cause the extended reality device 52 topresent the output representative data determined at block 506. That is,in response to determining the output representative data to bepresented, the processing system 62 may cause the extended realitydevice 52 to present such output representative data. In otherembodiments, the processing system 62 may provide the outputrepresentative data to the extended reality device 52, and the extendedreality device 52 may cause the output representative data to bepresented (e.g., via the display 120). The output representative datamay include, but is not limited to, visual data, audio data, hapticdata, gustatory data (e.g., a taste or flavor), and/or olfactory dataassociated with an environment, location data associated with a device(e.g., a component of the industrial automation system 72), andbiometric data associated with one or more users.

In this manner, the processing circuitry may output representative dataand cause the extended reality device 52 to present the retrieved inputdata and/or output data to a user, thereby simulating a previous,current, or potentially future status of the industrial automationsystem 72. The processing circuitry may also adjust or adapt userexperiences (i.e., determine new output representative data and providethe new output representative data) based on additional input receivedfrom the extended reality device 52, sensor(s) 78, or both. For example,the processing system 62 may process historical input data to simulatesimilar or related situations (e.g., to be interacted with in thecontext of a past event or as the basis for a future event to beexperienced). After providing corresponding output representative datato the extended reality device 52, the processing system 62 may receiveinput data from the extended reality device 52 associated with a firstoperational status (e.g., during a first operational mode of theindustrial automation component 74) of the industrial automation system72, such as a status that caused a maintenance task or operation to beperformed. In response, the processing system 62 may determine outputdata to be presented based on such input data and associate the outputdata with the first operational status of the industrial automationsystem 72.

At a later time, the processing system 62 may simulate the firstoperational status of the industrial automation system 72, such as totrain a remote user 82 to perform the maintenance ask, by presenting thestored output data associated with the first operational status of theindustrial automation system 72. Accordingly, while a user may select toexperience a past or current event, the processing system 62 maysimulate a potential future event by determining and providingadditional output representative data based on inputs received from theextended reality device 52. In other words, as a user interacts withoutput representative data (e.g., participate in a virtual or augmentedreality experience), the output representative data may be updated inresponse to, and based on, the user's interactions. The processingsystem 62 may also be used for applications other than training. Forinstance, the first operational status (or any past, current, of futurestatus) of the industrial automation system 72 (or a component thereof)may be simulated to investigate an event or incident within theindustrial automation system to determine the cause (or a potentialcause) of the incident. The event or incident may include, but is notlimited to, a variety of situations, such as a breakdown or otherinterruption to the operation of the industrial operation system 72 (ora component thereof). As another example, the processing system 62 maydetermine (and cause to be presented) output representative data toenable a user to review procedures within the industrial automationsystem 72. The procedures may include the general operation ormaintenance of the industrial automation system 72 as well as thecomponents thereof. The procedures may also relate to users or people(e.g., technicians) within the industrial automation system 72, such ashow the technicians operate equipment in the industrial automationsystem 72 or respond to events within the industrial automation system72.

The method 500 may be performed in conjunction with the techniquesdescribed above with respect to FIGS. 3-14 . As a nonlimiting example,after performing operations associated with block 508, the processingsystem 62 may provide or adjust additional output representative databased on an intent predicted by the processing system or user feedbackmade using the extended reality device 52. As such, in otherembodiments, the method 500 may include any operation or combination ofoperations included in the methods 140, 170, 200, 230, 260, 380, 440,470.

Furthermore, it should be noted that the output representative datagenerated while performing the method 500 may be modified based on auser's (past or present) interaction with output representative data,information about the user, information about the industrial automationsystem 72, or any combination thereof, thereby enabling the generationand editing of simulation content. For example, when performing asimulation for a user, the processing system 62 may utilize informationabout the user (e.g., based on a profile for the user, a role of theuser, and past activities of the user) to generate output representativedata specific to that user. As a more specific example, in the case of asimulation for a repair to a component of the industrial automationsystem 72, the processing system 62 may determine that a first user hasnot performed the repair recently (e.g., more than a threshold amount oftime has passed since the user last made a similar type of repair), thatthe first user is relatively inexperienced (e.g., based on an amount ofexperience indicated by the first user's profile being less than athreshold amount of time), or both. In response to such a determination,the processing system may generate output representative data thatincludes elements (e.g., a QR code or other visual representations) thatthe user can interact with to access a manual associated with performingthe repair, a service record of the component, a video tutorial forperforming the repair, or a combination thereof.

As another example, for a relatively more experienced second user whohas more recently completed the repair, the processing system 62 mayinclude fewer elements in the output representative data. For instance,if the first user were presented with elements for the manual, servicerecord, and video tutorial, the processing system 62 may include theelements for the manual and the service record because the second userdid not access the video tutorial when last performing the repair. Theprocessing system 62 may monitor how a user interacts with the outputrepresentative data. Continuing with the example of the second user,when performing the repair in the present, the second user may provideuser input to access the service record upon being presented with theoutput representative data. The processing system 62 may record such anaction, and when the second user subsequently performs a differentrepair, the output representative data may automatically include theservice record. That is, the service record may be presented to thesecond user without the second user providing user input to display theservice record.

As a further example, while interacting with the output representativedata, the operator may verbally complain that instructions for therepair that are being displayed are obscuring the user's view. Inresponse to such a cue (e.g., alone or in combination with user input,such as user input to move where the instructions are displayed), theprocessing system 62 may automatically generate output representativedata in the future that includes in the instructions in a locationacceptable to the second user. Furthermore, based on data collected frommultiple users, the processing system 62 may alter baseline or defaultoutput representative data. For example, if the processing system 62detects that users rarely interact with the element for manuals, theprocessing system 62 may remove the element from the outputrepresentative data or replace the element with an element associatedwith other content (e.g., an element for content that the processingsystem 62 determines more users have accessed). As such, the processingsystem 62 may automatically generate and alter output representativedata that is both user-specific and context-specific.

Accordingly, by performing the method 500, the processing system 62enables users to experience fully immersive scenarios (e.g., simulationsinvolving using virtual reality, augmented reality, or both) that may bepast events, current events, or events that could potentially occur at afuture time. In this manner, the processing system 62 may be used for avariety of applications, such as to train technicians, engineers, orother users how to perform various tasks associated with the industrialautomation system 72.

Embodiments of the present disclosure improve data presentation to auser via an extended reality device. For example, a processing systemmay receive input data (e.g., input sensory data) associated with anindustrial automation system, categorize the input data, prioritize thecategorized input data based on context information, and cause theextended reality device to present output representative data accordingto the priority. In particular, the processing system may prioritize thecategorized input data based on a relevancy or importance of the inputdata, such as that determined using context information associated withthe user and/or the industrial automation system. In this manner, theprocessing system may better assist the user perform a desired task,such as by providing sufficient information to the user withoutoverwhelming the user. Indeed, the processing system may cause theextended reality device to present output representative data in aparticular format based on the priority and the input data. As a result,the processing system may improve the manner in which variousinformation is presented to the user, such as by presenting visual data,audio data, haptic feedback, and the like, in a more intuitive manner.

While only certain features of the disclosure have been illustrated anddescribed herein, many modifications and changes will occur to thoseskilled in the art. It is, therefore, to be understood that the appendedclaims are intended to cover all such modifications and changes as fallwithin the true spirit of the disclosure.

The techniques presented and claimed herein are referenced and appliedto material objects and concrete examples of a practical nature thatdemonstrably improve the present technical field and, as such, are notabstract, intangible or purely theoretical.

Further, if any claims appended to the end of this specification containone or more elements designated as “means for [perform]ing [a function]. . . ” or “step for [perform]ing [a function] . . . ”, it is intendedthat such elements are to be interpreted under 35 U.S.C. 112(f).However, for any claims containing elements designated in any othermanner, it is intended that such elements are not to be interpretedunder 35 U.S.C. 112(f).

1. A non-transitory computer-readable medium comprising instructions,wherein the instructions, when executed by processing circuitry, areconfigured to cause the processing circuitry to perform operationscomprising: receiving a plurality of sensory datasets associated with anindustrial automation system from a plurality of sensors, wherein theplurality of sensors comprises an image sensor, a microphone, a hapticsensor, a movement sensor, a biometric sensor, an odor sensor, or anycombination thereof; receiving positioning data via an extended realitydevice associated with a user; determining a first virtual positioningof the user in a virtual coordinate system based on the positioningdata; determining a second virtual positioning of an industrialautomation system in the virtual coordinate system based on theplurality of sensory datasets; determining output representative data tobe presented by the extended reality device based on the plurality ofsensory datasets and in accordance to the first virtual positioningrelative to the second virtual positioning; and instructing the extendedreality device to present the output representative data.
 2. Thenon-transitory computer-readable medium of claim 1, wherein theinstructions, when executed by the processing circuitry, are configuredto cause the processing circuitry to perform operations comprising:receiving updated positioning data via the extended reality device;determining adjustment of the user from the first virtual positioning toa third virtual positioning in the virtual coordinate system based onthe updated positioning data; determining additional outputrepresentative data to be presented by the extended reality device basedon the plurality of sensory datasets and in accordance to the thirdvirtual positioning relative to the second virtual positioning; andinstructing the extended reality device to present the additional outputrepresentative data.
 3. The non-transitory computer-readable medium ofclaim 1, wherein the instructions, when executed by the processingcircuitry, are configured to cause the processing circuitry to performoperations comprising: determining a third virtual positioning of anadditional user in the virtual coordinate system; determining additionaloutput representative data to be presented by the extended realitydevice based on the plurality of sensory datasets and in accordance tothe third virtual positioning relative to the first virtual positioning;and instructing the extended reality device to present the additionaloutput representative data.
 4. The non-transitory computer-readablemedium of claim 3, wherein the instructions, when executed by theprocessing circuitry, are configured to cause the processing circuitryto perform operations comprising: receiving audio data associated withthe additional user via an additional extended reality device; andinstructing the extended reality device to present an audio output basedon the audio data and in accordance to the third virtual positioningrelative to the first virtual positioning.
 5. The non-transitorycomputer-readable medium of claim 3, wherein the instructions, whenexecuted by the processing circuitry, are configured to cause theprocessing circuitry to perform operations comprising: detecting thatthe first virtual positioning and the third virtual positioningintersect with one another; and instructing the extended reality deviceto present feedback in response to detecting that the first virtualpositioning and the third virtual positioning intersect with oneanother.
 6. The non-transitory computer-readable medium of claim 5,wherein the instructions, when executed by the processing circuitry, areconfigured to cause the processing circuitry to perform operationscomprising: determining a location on the first virtual positioningwhere the first virtual positioning and the third virtual positioningintersect with one another; and instructing the extended reality deviceto present the feedback based on the location on the first virtualpositioning.
 7. The non-transitory computer-readable medium of claim 6,wherein the extended reality device comprises a plurality of components,each component of the plurality of components is configured to presenthaptic feedback, and the instructions, when executed by processingcircuitry, are configured to cause the processing circuitry to performoperations comprising: determining a subset of the plurality ofcomponents corresponding to the location on the first virtualpositioning; and instructing the subset of the plurality of componentsto present the haptic feedback.
 8. The non-transitory computer-readablemedium of claim 1, wherein the instructions, when executed by theprocessing circuitry, are configured to cause the processing circuitryto instruct the extended reality device to perform operationscomprising: determining a third virtual positioning of an additionaluser in the virtual coordinate system; receiving a request to present avisual image representative of the additional user via the extendedreality device; and instructing the extended reality device to presentthe visual image in accordance to the third virtual positioning relativeto the first virtual positioning.
 9. A non-transitory computer-readablemedium comprising instructions, wherein the instructions, when executedby processing circuitry, are configured to cause the processingcircuitry to perform operations comprising: receiving a plurality ofsensory datasets associated with an industrial automation system from aplurality of sensors, wherein the plurality of sensors comprises animage sensor, a microphone, a haptic sensor, a movement sensor, abiometric sensor, an odor sensor, or any combination thereof; monitoringpositioning data received via an extended reality device associated witha user; determining a first virtual positioning of the user in a virtualcoordinate system based on the positioning data; comparing the firstvirtual positioning to a second virtual positioning of an industrialautomation system in the virtual coordinate system; determining a subsetof the plurality of sensory datasets based on the first virtualpositioning relative to the second virtual positioning; determiningoutput representative data to be presented by the extended realitydevice based on the subset of the plurality of sensory datasets; andinstructing the extended reality device to present the outputrepresentative data.
 10. The non-transitory computer-readable medium ofclaim 9, wherein the instructions, when executed by the processingcircuitry, are configured to cause the processing circuitry to performoperations comprising: monitoring additional positioning data receivedvia an additional extended reality device associated with the additionaluser; determining a third virtual positioning of the additional user inthe virtual coordinate system based on the additional positioning data;comparing the third virtual positioning to the second virtualpositioning of the industrial automation system in the virtualcoordinate system; determining an additional subset of the plurality ofsensory datasets based on the third virtual positioning relative to thesecond virtual positioning; determining additional output representativedata to be presented by the additional extended reality device based onthe additional subset of the plurality of sensory datasets; andinstructing the additional extended reality device to present theadditional output representative data.
 11. The non-transitorycomputer-readable medium of claim 10, wherein the instructions, whenexecuted by the processing circuitry, are configured to cause theprocessing circuitry to perform operations comprising: receiving a userinput indicative of a selection of the additional user via the extendedreality device; and instructing the extended reality device to presentthe additional output representative data in response to receiving theselection of the additional user.
 12. The non-transitorycomputer-readable medium of claim 9, wherein the instructions, whenexecuted by the processing circuitry, are configured to cause theprocessing circuitry to perform operations comprising: determining athird virtual positioning of an object in the virtual coordinate systemvia an additional extended reality device; comparing the first virtualpositioning of the user to the third virtual positioning of the object;and instructing the extended reality device to present feedback based onthe first virtual positioning relative to the third virtual positioning.13. The non-transitory computer-readable medium of claim 12, wherein theinstructions, when executed by the processing circuitry, are configuredto cause the processing circuitry to perform operations comprising:detecting that the first virtual positioning and the third virtualpositioning intersect with one another; and instructing the extendedreality device to present the feedback in response to detecting that thefirst virtual positioning and the third virtual positioning intersectwith one another.
 14. The non-transitory computer-readable medium ofclaim 10, wherein the instructions, when executed by the processingcircuitry, are configured to cause the processing circuitry to performoperations comprising: instructing the extended reality device topresent a visual image representative of the additional user based onthe third virtual positioning of the additional user relative to thefirst virtual positioning of the user; and instructing the extendedreality device to adjust the visual image based on the additionalpositioning data received via the additional extended reality device.15. The non-transitory computer-readable medium of claim 9, wherein theinstructions, when executed by the processing circuitry, are configuredto cause the processing circuitry to perform operations comprising:receiving an input indicative of a selection of a third virtualpositioning of the user in the virtual coordinate system; determining anadditional subset of the plurality of sensory datasets based on thethird virtual positioning relative to the second virtual positioning;determining additional output representative data to be presented by theextended reality device based on the additional subset of the pluralityof sensory datasets; and instructing the extended reality device topresent the additional output representative data.
 16. A method,comprising: receiving, via processing circuitry, a plurality of sensorydatasets associated with an industrial automation system from aplurality of sensors, wherein the plurality of sensors comprises animage sensor, a microphone, a haptic sensor, a movement sensor, abiometric sensor, an odor sensor, or any combination thereof; receiving,via the processing circuitry, a first input indicative of a firstvirtual positioning of a user in a virtual coordinate system;determining, via the processing circuitry, first output representativedata to be presented by an extended reality device based on theplurality of sensory datasets and in accordance to the first virtualpositioning of the user relative to a second virtual positioning of anindustrial automation system in the virtual coordinate system;instructing, via the processing circuitry, the extended reality deviceto present the first output representative data; receiving, via theprocessing circuitry, a second input indicative of an adjustment of theuser from the first virtual positioning to a third virtual positioningrelative to the second virtual positioning of the industrial automationsystem in the virtual coordinate system; determining, via the processingcircuitry, second output representative data to be presented by theextended reality device based on the plurality of sensory datasets andin accordance to the third virtual positioning of the user relative tothe second virtual positioning of the industrial automation system; andinstructing, via the processing circuitry, the extended reality deviceto present the second output representative data.
 17. The method ofclaim 16, wherein the first input, the second input, or both, comprisesany combination of a user input or sensor data received from theextended reality device.
 18. The method of claim 16, comprisingreceiving, via the processing circuitry, a third input indicative of afourth virtual positioning of an additional user in the virtualcoordinate system, wherein each of the first output representative dataand the second output representative data comprises image datacomprising the additional user and the industrial automation systempositioned relative to one another in accordance to the fourth virtualpositioning of the additional user relative to the second virtualpositioning relative of the industrial automation system.
 19. The methodof claim 18, wherein the first output representative data comprisesfirst image data comprising the additional user and the industrialautomation system presented in accordance to the first virtualpositioning of the user relative to the second virtual positioning ofthe industrial automation system and the fourth virtual positioning ofthe additional user, and the second output representative data comprisessecond image data comprising the additional user and the industrialautomation system presented in accordance to the third virtualpositioning of the user relative to the second virtual positioning ofthe industrial automation system and the fourth virtual positioning ofthe additional user.
 20. The method of claim 16, comprising:identifying, via the processing circuitry, a component of the industrialautomation system based on the first virtual positioning of the userrelative to the second virtual positioning of the industrial automationsystem, wherein the component is within a field of view of the user,within a threshold distance from the user, or both; identifying, via theprocessing circuitry, information associated with the component; andinstructing, via the processing circuitry, the extended reality deviceto present the information associated with the component.